Why is a high-level deployment infrastructure important? Deployment is something that should be simple, accessible, and repeatable. It should be as close to a "single click" as possible. Previously, for me, it was a bash script that exported some SVN branches. While this worked fine, as projects progress, you want some accountability, history, and the ability to roll back mission critical applications when something goes wrong with a deploy.

Capistrano is an open source, command line, deployment tool that provides all of these features. It's written in Ruby. You leverage a variety of built in "recipes" (Capistrano's term for a deployment script) that execute certain procedures to deploy an app. Out-of-the-box it's ideally built to deploy a Rails app. However, after some minor tweaks it can deploy most anything and do it well. It can restart servers, update symlinks, change permissions - pretty much anything. It assumes you access your POSIX compliant server via SSH via the same password (or have ssh keys setup).

Webistrano is an open source web front-end for Capistrano. It's a convenience layer that abstracts the command line away and provides an interface to perform the same tasks. This interface shows history as well as providing a convenient GUI for creating new deployment projects, stages, and recipes. Highly recommended.

Let's get down to business. This post makes a few assumptions about things you've already installed and used previously.

• Ruby 1.8.5+
• RubyGems
• MySQL 5.0+
• Apache 2+
• Subversion and a repository containing the "production" branch of your app.

Installing Capistrano

Well, this is an easy one (you probably want to do this as root):

gem install capistrano

Installing Webistrano

Also fairly easy, with a little splash of configuration.

# wget http://labs.peritor.com/webistrano/attachment/wiki/Download/webistrano-1.4.zip
# unzip webistrano-1.4.zip
# mv webistrano-1.4 /path/to/where/you/want/webistrano

Setup the database tables and create a new webistrano user (obviously be conscious of your security preferences for access to your database in the host and password portions):

# mysql
mysql> CREATE DATABASE `webistrano`;
mysql> CREATE USER 'webistrano'@'localhost' IDENTIFIED BY 'password';
mysql> GRANT ALL PRIVILEGES ON `webistrano`.* TO 'webistrano'@'localhost' WITH GRANT OPTION;

Now, in the directory where you placed webistrano you're going to want to copy config/database.yml.sample to config/database.yml. Edit this file, in the production area, to match your database settings. By default the file expects a socket to connect, you can chase this by specifying host: and port:. (Keep in mind Webistrano is simply a Rails app).

You should now be able to have Rails migrate the new database you created. In the webistrano directory:

# RAILS_ENV=production rake db:migrate

Finally, copy config/webistrano_config.rb.sample to config/webistrano_config.rb and edit according to your preferred mail settings.

We can now test to see if webistrano is working properly by serving it via mongrel:

# ruby script/server -d -e production -p 3000

This starts a single mongrel daemon, using the production environment, listening on port 3000. You should now be able to hit http://127.0.0.1:3000/ and get the Webistrano login prompt. If this is working, kill that mongrel instance.

For longer term serving I decided to go with Phusion Passenger (essentially mod_rails for Apache). It's a nearly zero configuration solution for serving a rails app and will feel at home to anyone with experience serving PHP apps via Apache and mod_php.

Installing Phusion Passenger

Again, as root:

# gem install passenger
# passenger-install-apache2-module

The second command will invoke an installer which compiled Passenger and provides instructions on integrating it into your Apache config. Essentially, edit your httpd.conf as follows (these were specific to my install, make sure to use the ones provide by the installer for you):

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.9
PassengerRuby /usr/bin/ruby

Now you can simply add VirtualHost entries to your httpd.conf for any of your Rails apps. Let's add one for Webistrano:

<VirtualHost *:80>
ServerName webistrano.mydomain.com
DocumentRoot /path/to/webistrano/public
</VirtualHost>

Yes, Passenger makes it that simple. Add configuration directives as needed for your environment.

Now Webistrano should be serving from the VirtualHost you specified, seamlessly, via Passenger.

Deploying A Non-Rails App

Now the fun stuff.

Capistrano breaks things down into projects, stages, and recipes. Each app you want managed by capistrano should be it's own project. Each project should have a stage for at least production and optionally staging and development.

Hosts are added globally and form the targets of a deploy for any given project. Hosts can include web, app, and database servers.

Deployments in Capistrano are done to a child directory under "releases" named via the date and time of the deployment. By default 5 releases are kept and available to rollback to. Upon successful deployment a symlink (default is called "current" and can be modified via the current_path configuration variable) is updated to that release directory. It is this symlink that should be targeted by your webserver (your DocumentRoot in Apache).

Capistrano also creates a "shared" directory that is symlinked to in each release useful for storing logs and other data that should be maintained through each deployment.

For non-rails apps you'll use the "Pure File" project type when creating your new project. Upon project creation you can add configuration variables specific to your project. I recommend using :export instead of :checkout for deploy_via for production subversion deployments as this doesn't expose .svn directories. Use an SSH user that has enough permissions to create directories where your deploy will occur or, specify use_sudo to true and create a new configuration variable admin_runner and set it to the same user as runner.

Add a stage to your new project for "production". In the "Manage Hosts" page add a new host for each of your application servers. Then add each host as a target of your "production" stage of your project.

At this point you should be able to execute the "Setup" task for your "production" stage. This is a one time task that simply creates the directories.

Assuming this went successfully, try doing a "Deploy" and see if that finishes without error. You might have to play around with permissions and other minor issues - post a comment if you have any specific questions.

For my PHP framework there are a couple specific tasks I wanted to run in addition to the default Capistrano tasks. You do this by creating custom recipes in the "Manage Recipes" page in Webistrano. Recipes are simply procedures written in ruby. Here's what my recipe looks like:

namespace :deploy do
	task :setup, :except => { :no_release => true } do
		dirs = [deploy_to, releases_path, shared_path]
		dirs += shared_children.map { |d| File.join(shared_path, d) }
		run "#{try_sudo} mkdir -p #{dirs.join(' ')} && #{try_sudo} chmod g+w #{dirs.join(' ')}"
		run "chmod 777 #{shared_path}/log"
	end

	task :finalize_update, :except => { :no_release => true } do
		run "mkdir -p #{latest_release}/app/tmp"
		run "chmod -R 777 #{latest_release}/app/tmp"
		run "rm -rf #{latest_release}/app/logs"
		run "ln -s #{shared_path}/log #{latest_release}/app/logs"
		run "cp #{latest_release}/public_html/.htaccess-production #{latest_release}/public_html/.htaccess"
		run "cp #{latest_release}/app/config/config-production.php #{latest_release}/app/config/config.php"
		run "cp #{latest_release}/app/config/db-default.php #{latest_release}/app/config/db.php"
		run "cp #{latest_release}/app/config/memcache-default.php #{latest_release}/app/config/memcache.php"
	end
end

If you're not familiar with Ruby - what this code is essentially doing is overwriting two tasks in the :deploy namespace with my custom code.

The first, :setup, simply duplicates the base :setup functionality discussed above (creating the releases and shared directories) and chmods the shared log directory to be writable.

The second, :finalize_update, performs a variety of configuration tasks for a PHP app built with my framework. Also, you'll notice that I'm removing my app's logs directory and symlinking to the shared log directory. This way all releases will log to the same directory, consistently.

In my case all of these procedures are command line instructions. Alternatively, you can do a variety of things leveraging the full breadth of the Ruby language and any gem you'd like to introduce. Things such as accessing your CDN API to clear image, JS, or CSS caching, etc.

Deploying Django Apps

First off it's worth noting that I serve my Django apps via mod_wsgi. To make the deployment process easier here's what my app.wsgi script looks like:

import os
import sys

appdir = os.path.normpath(os.path.join(os.path.realpath(os.path.dirname(__file__)), '..'))
sys.path.insert(0, appdir)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
os.environ['PYTHON_EGG_CACHE'] = os.path.join(appdir, '.python-eggs')
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

This code allows us to avoid having to hardcode paths in the wsgi script (and thus avoid having to change them when we deploy). It assumes the following directory structure:

.python-eggs (egg cache)
apps (apps path is added to python system path in settings.py)
public (where your .wsgi script resides)
site_media
templates
settings.py
settings-production.py (used for deploy)
urls.py
...

If you follow this convention, the following Capistrano recipe works great:

namespace :deploy do
	task :setup, :except => { :no_release => true } do
		dirs = [deploy_to, releases_path, shared_path]
		dirs += shared_children.map { |d| File.join(shared_path, d) }
		run "#{try_sudo} mkdir -p #{dirs.join(' ')} && #{try_sudo} chmod g+w #{dirs.join(' ')}"
		run "chmod 777 #{shared_path}/log"
	end

	task :finalize_update, :except => { :no_release => true } do
		run "rm -rf #{latest_release}/logs"
		run "ln -s #{shared_path}/log #{latest_release}/logs"
		run "cp #{latest_release}/settings-production.py #{latest_release}/settings.py"
		run "mkdir -p #{latest_release}/.python-eggs"
		run "chmod 777 #{latest_release}/.python-eggs"
	end
end

Fin

This should give you a nice intro to leveraging Capistrano via Webistrano. Feel free to comment with questions, suggestions, or anything else!

Related posts:

1. Installing Ruby Enterprise Edition with Phusion Passenger
2. 8 Books To Get A Developer For The Holidays
3. Be Language Agnostic – Solve the Problem!

The Pragmatic Programmer: From Journeyman to Master

Write better, cleaner, more maintainable code. Learn how to manage your projects and focus on shipping your product. With insight that covers the gamut of software development from low level to management
this one is a must have for anyone involved in this industry.

The Passionate Programmer: Creating a Remarkable Career in Software Development

Highly recommended! Read my full review.

Code Complete: A Practical Handbook of Software Construction

Another classic "software construction" book. Sharpen your saw with timeless information that can be applied to any project in any language. Less bugs, more productivity, more programmer happiness.

Coders at Work

This one is different. Written as a set of interview transcripts with 15 legendary industry giants, this book is a fantastic insight into how some of the great minds think. It's inspiring to hear it from the source, must have!

Programming Clojure

A developer should learn at least one new language a year. This year that language should be Clojure. Clojure is a dynamic, general purpose, language targeting the Java virtual machine and designed for multi-threaded use. It's growing popularity, ability to leverage the Java standard library, and its multi-threaded nature make this a must have.

The Mythical Man-Month: Essays on Software Engineering

Another classic. Primarily discusses project management from the perspective of Fred Brooks and his experiences at IBM. Brooks' Law states that "adding manpower to a late software project makes it later".

Don't Make Me Think: A Common Sense Approach to Web Usability

Web developers should always keep in mind the user of the product their creating. Usability becomes increasingly important as applications move to the web. The design and usability of your app can make or break its success. This classic is a must read.

Design Patterns: Elements of Reusable Object-Oriented Software

This classic known most commonly as the "gang of four" book is the definitive reference on design patterns. Covering all of the most common cases and time and time again serving as an invaluable source of information.

Related posts:

1. Be Language Agnostic – Solve the Problem!
2. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger
3. Django 1.0 Template Development: Sample Chapter “Serving Multiple Templates”

Methodology: I would first need a good Twitter gem for Ruby, and I'd need to decide which URL shortening service I'd use.  Ruby Twitter seemed to be the simplest gem for Twitter.  For the URL shortening I chose bit.ly, because the bitly gem seemed like the easiest, and the documentation was good.  My plan of attack was:

1. Grab one item from the queue, that has not been Twittered
2. Shorten the URL via bitly
3. Send the item's title and shortened URL to Twitter
4. Save the shortened URL in the database so I could retrieve stats later

The script that did this would be run every hour.

Implementation: The two gems made this an almost trivial implementation.

require 'twitter'
require 'bitly'

@item = Item.find(:first, :conditions => "send_to_twitter = 1 and twitterd = 0", :order => "posted_on desc")

if !@item.nil?
   b = Bitly.new(username, password)
   @url = b.shorten("http://news.sensusdivinitatis.com/item/#{@item.id}").short_url

   httpauth = Twitter::HTTPAuth.new(username, password)
   base = Twitter::Base.new(httpauth)
   base.update("#{@item.title[0...110]} - #{@url}") #shorten the title if it's too long

   Item.update(@item.id, :twitterd => 1, :bitly_url => @url) #save the bit.ly url
end

That's pretty much it.  Incidentally, the bitly gem makes it very easy to grab the stats for any URL.  For instance, if you wanted to see how many clicks a given URL has received:

require 'twitter'
require 'bitly'

@item = Item.find(id)
b = Bitly.new(username, password)
@clicks = b.stats(i.bitly_url).stats["clicks"]

Done.

Related posts:

1. Flickr, RSS, and Ruby
2. Be Language Agnostic – Solve the Problem!
3. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger

Watch this video. It's excellent food for thought and drives home the importance, as a programmer, of learning new, different, languages to expand your ability to solve problems in a variety of ways. The video mentions the Sapir–Whorf hypothesis which suggests that a particular language influences how a person understands and interacts with the world. This makes a lot of sense.

A programmer who thinks only in terms of a single language will attempt to solve every problem with that language. You need more than one tool on your belt because not every problem is a nail.

Related posts:

1. 8 Books To Get A Developer For The Holidays
2. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger
3. Django and Python First Impressions – Part II

It might be helpful to note which books available today cover Django 1.0. I do realize that the "official" Django book covering 1.0 is in the works, but, in the meantime I recommend Python Web Development with Django (Developer's Library) and Pro Django (Expert's Voice in Web Development).

Lets start with the template system. While Django's template system is powerful, I'm sure, it's basically a free for all of file names and directory structures. You could conceivably create a single 'templates' directory and place all of a projects template files in that directory (naming the files as you please) for any and all the apps that compose the project. While it's not recommended you actually do that, Django would support it because it doesn't seem to really dictate or enforce any particular convention. I feel like Rails, comparatively speaking, provides solid conventions for a developer to follow in terms of file naming, directory naming, and directory structure.

I was also surprised at the fact that Django does NOT bundle a pluralization library. Even simple cases (ending in y to ies) aren't handled automatically. A model named 'Category' appears as 'Categorys'. It does provide support for explicitly specifying plural names for Models via the verbose_name_plural Meta property:

class Category(models.Model):
    name = models.CharField(max_length=128)
    class Meta:
        verbose_name_plural = 'categories'

Coming from a world of a custom built PHP framework - it's REALLY nice to have features like the command line sandbox to be able to play with Models and interact with the project's codebase. It's also great that Django bundles a development webserver for a quick and easy create-test-edit cycle. It recognizes changes to your code while it's running - there's no need to restart manually.

The effects are immeasurable with respect to the fact that Python functions are first-class. Django uses this extensively (such as your URL configuration or default values for Models). It's extremely intuitive to be able to do the following in your Model definition:

class Post(models.model)
    stamp = models.DateTimeField(default=datetime.now)

Notice this wasn't written as datetime.now(). That would actually execute the function when the class is declared and all entries would receive an identical stamp. Instead we're passing a reference to the function. Django detects this and calls the function when the Model is instantiated.

I'm also excited about the way the Django framework handles requests and responses. The concept of the view receiving an HttpRequest object, giving context, and returning an HttpResponse object just makes sense. It's very much in line with the way HTTP works. It's simple, powerful, and elegant.

I've read about (and watched video of) some issues that people have with Django. These usually refer to difficulties one may have scaling it. Lack of built-in support for multiple databases and sharding are cited, among other reasons. I think, for what it's able to do right out of the box, it's fantastic. Scaling (in general) isn't always straightforward. In many cases it requires specific tools and solutions for the task at hand. These issues should in no way prevent you from using Django for a project!

More soon.

Related posts:

1. Getting Started with Django and Python – First Impressions
2. Adventures in Django and Python – Part III
3. Django 1.1 Released

Before you do anything, and to save you trouble down the road, I recommend generating a list of all the gems currently installed. You'll need this later because you'll have to reinstall all of them in Ruby Enterprise Edition.

$ gem list

*** LOCAL GEMS ***

actionmailer (2.0.2)
actionpack (2.0.2)
activerecord (2.0.2)
activeresource (2.0.2)
activesupport (2.0.2)
acts_as_taggable (2.0.2)
amazon-ecs (0.5.3)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
fastthread (1.0.1)
gem_plugin (0.2.3)
hpricot (0.6)
mongrel (1.1.4)
mysql (2.7)
passenger (1.0.1)
rails (2.0.2)
rake (0.8.1)
rspec (1.1.3)
will_paginate (2.2.2)
xml-simple (1.0.11)

Copy your list and save it somewhere. Next, we download and extract the current version of Ruby Enterprise Edition. The latest version number can be found at the download page.

$ wget http://rubyforge.org/frs/download.php/48623/ruby-enterprise-1.8.6-20081215.tar.gz
$ tar xzvf ruby-enterprise-1.8.6-20081215.tar.gz

And run the installer

$ ./ruby-enterprise-1.8.6-20081215/installer

As the installation proceeds, you'll be prompted for an installation directory. I went with the default. When the installation is over, you should see a message like this:

Ruby Enterprise Edition is successfully installed!
If want to use Phusion Passenger (http://www.modrails.com) in combination
with Ruby Enterprise Edition, then you must reinstall Phusion Passenger against
Ruby Enterprise Edition, as follows:

/opt/ruby-enterprise-1.8.6-20081215/bin/passenger-install-apache2-module

Make sure you don't forget to paste the Apache configuration directives that
the installer gives you.

If you ever want to uninstall Ruby Enterprise Edition, simply remove this
directory:

/opt/ruby-enterprise-1.8.6-20081215

If you have any questions, feel free to visit our website:

http://www.rubyenterpriseedition.com

Enjoy Ruby Enterprise Edition, a product of Phusion (www.phusion.nl)

I do want to use Ruby Enterprise Edition and Apache, so I immediately installed the Apache module.

$ /opt/ruby-enterprise-1.8.6-20081215/bin/passenger-install-apache2-module

You need to pay attention to the Apache directives that this installation generates, since you'll have to put them into your httpd.conf:

The Apache 2 module was successfully installed.

Please edit your Apache configuration file, and add these lines:

LoadModule passenger_module /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/passenger-2.0.6/ext/apache2/mod_passenger.so
PassengerRoot /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/passenger-2.0.6
PassengerRuby /opt/ruby-enterprise-1.8.6-20081215/bin/ruby

What that message doesn't explain is that, if you previously installed Passenger without Ruby Enterprise Edition, you need to comment out any old directives attached to that previous setup. I made that mistake the first time around.

Next you have to reinstall all the gems that you were using with old Ruby, including Rails:

/opt/ruby-enterprise-1.8.6-20081215/bin/ruby /opt/ruby-enterprise-1.8.6-20081215/bin/gem install rails --version 2.0.2

Just to prove my point, if you listed all of Enterprise Ruby's gems at this point, the list would look something like this:

$ /opt/ruby-enterprise-1.8.6-20081215/bin/ruby /opt/ruby-enterprise-1.8.6-20081215/bin/gem list

*** LOCAL GEMS ***

actionmailer (2.2.2, 2.0.2)
actionpack (2.2.2, 2.0.2)
activerecord (2.2.2, 2.0.2)
activeresource (2.2.2, 2.0.2)
activesupport (2.2.2, 2.0.2)
fastthread (1.0.1)
mysql (2.7)
passenger (2.0.6)
rack (0.9.1)
rails (2.2.2, 2.0.2)
rake (0.8.3)

None of my old gems are installed. Once Rails is installed, you need to reinstall your old gems:

$ /opt/ruby-enterprise-1.8.6-20081215/bin/ruby /opt/ruby-enterprise-1.8.6-20081215/bin/gem install will_paginate
$ /opt/ruby-enterprise-1.8.6-20081215/bin/ruby /opt/ruby-enterprise-1.8.6-20081215/bin/gem install xml-simple
$ /opt/ruby-enterprise-1.8.6-20081215/bin/ruby /opt/ruby-enterprise-1.8.6-20081215/bin/gem install hpricot
$ /opt/ruby-enterprise-1.8.6-20081215/bin/ruby /opt/ruby-enterprise-1.8.6-20081215/bin/gem install rmagick

And on and on. If you haven't used Passenger before, check out the Deployment Guide. If you have used Passenger and your Rails apps are already set up in Apache to take advantage of it's deployment procedures, simply restart Apache.

Related posts:

1. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger
2. Ruby On Rails and SliceHost Part 1: Initial Setup
3. Twitter and SD News

Change the default root password

>passwd

Update all yum packages

>yum update

Bring in some standard packages

>yum install wget openssl-devel lynx zlib zlib-devel vixie-cron curl lynx subversion make gcc automake

Install Apache

>yum install httpd

Setup your domain records
http://articles.slicehost.com/2007/10/24/creating-dns-records

Create your VirtualHosts in Apache's httpd.conf
http://httpd.apache.org/docs/2.2/vhosts
Note: Apache's conf directory will be /etc/httpd/conf. Don't forget to restart Apache after setting up each new vhost:

>service httpd restart

See http://mongrel.rubyforge.org/docs/apache.html for details on how to get Apache and mongrel to play with each other.

Start Apache

>httpd service start

Install MySQL

>yum install mysql mysql-server

See http://dev.mysql.com/doc/ for post-installation instructions (setting up initial accounts, etc.)

Install Ruby (1.8.6)

>wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p111.tar.gz
>gzip -d ruby-1.8.6-p111.tar.gz
>tar xvf ruby-1.8.6-p111.tar.gz
>cd ruby-1.8.6-p111.tar.gz
>./configure
>make
>make install

Install Ruby Gems (1.0.1)

>wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
>tar xvf rubygems-1.0.1.tgz
>cd rubygems-1.0.1.tgz
>ruby setup.rb

Install Rails (2.0.2)

>gem install rails --include-dependencies

Install Mongrel

>gem install mongrel

Optional for phpMyAdmin
Install php

>yum install php

Install php extensions necessary for phpMyAdmin

>yum install php-mcrypt
>yum install php-mbstring

Install phpmyadmin
http://www.phpmyadmin.net/documentation/#setup

Any comments or feedback is appreciated.

Related posts:

1. Installing Ruby Enterprise Edition with Phusion Passenger
2. Setup Python 2.6.4, mod_wsgi 2.6, and Django 1.1.1 on CentOS 5.3 (cPanel)
3. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger