I was investigating some intermittent issues in an application at work - transient DNS issues were causing the application to become unresponsive. This was confusing at first because it was written to perform HTTP requests asynchronously. The important thing here is that it was specifically DNS resolution that was failing. As I dug deeper I realized that simple health checks, ones that did not perform any HTTP requests, were also hanging... something had to be blocking.

Taking a look at the Tornado source, unsurprisingly, it was leveraging the multi functionality of libcurl (via pycurl). What was surprising to me was that the DNS resolution portion of the multi interface, by default, blocks on non-windows installations. Only when compiled with c-ares support does it perform these async.

You learn something new every day!

Related posts:

1. Python’s Tornado has swept me off my feet
2. Tornado 1.0 Released
3. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger

Tornado is a non-blocking web server written in Python. It's structure is similar to web.py so users of that popular Python web framework will feel right at home. This is a structure that lends itself really well to developing RESTful APIs as the methods you write to handle incoming requests are named after the HTTP methods used:

class PlaceHandler(tornado.web.RequestHandler):
    def get(self, id):
        # respond to a GET
        self.write('GETting something')

    def post(self):
        # respond to a POST
        self.write('POSTing something')

You match URI paths to "handlers" (the controller for those MVC folk) via a list of regex, handler tuples that instantiate an "application".

application = tornado.web.Application([
    (r"/place", PlaceHandler),
    (r"/place/([0-9]+)", PlaceHandler)
])

if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

As usual any values that are captured from the regex are passed, in order, to the method that receives the request in the handler.

Because of it's non-blocking nature Tornado bundles an asynchronous HTTP client for use internally. Additional modules include a command line and config file convenience library, escaping, 3rd party authentication (Facebook, Twitter, etc.), a wrapper around MySQLdb, and templating. All in all this makes it a formidable web framework in its own right, especially if you're looking for something that's light and FAST.

In production, I'm running 4 Tornado instances per server behind nginx.

One issue not addressed out of the box was daemonizing the Tornado instance. I added PID file management and the ability to daemonize as follows (pid.py module follows):

# capture stdout/err in logfile
log_file = 'tornado.%s.log' % options.port
log = open(os.path.join(settings.log_path, log_file), 'a+')

# check pidfile
pidfile_path = settings.PIDFILE_PATH % options.port
pid.check(pidfile_path)

# daemonize
daemon_context = daemon.DaemonContext(stdout=log, stderr=log, working_directory='.')
with daemon_context:
    # write the pidfile
    pid.write(pidfile_path)

    # initialize the application
    http_server = tornado.httpserver.HTTPServer(application.app)
    http_server.listen(options.port, '127.0.0.1')

    try:
        # enter the Tornado IO loop
        tornado.ioloop.IOLoop.instance().start()
    finally:
        # ensure we remove the pidfile
        pid.remove(pidfile_path)

And now the pid.py module:

# pid.py - module to help manage PID files
import os
import logging
import fcntl
import errno

def check(path):
    # try to read the pid from the pidfile
    try:
        logging.info("Checking pidfile '%s'", path)
        pid = int(open(path).read().strip())
    except IOError, (code, text):
        pid = None
        # re-raise if the error wasn't "No such file or directory"
        if code != errno.ENOENT:
            raise

    # try to kill the process
    try:
        if pid is not None:
            logging.info("Killing PID %s", pid)
            os.kill(pid, 9)
    except OSError, (code, text):
        # re-raise if the error wasn't "No such process"
        if code != errno.ESRCH:
            raise

def write(path):
    try:
        pid = os.getpid()
        pidfile = open(path, 'wb')
        # get a non-blocking exclusive lock
        fcntl.flock(pidfile.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
        # clear out the file
        pidfile.seek(0)
        pidfile.truncate(0)
        # write the pid
        pidfile.write(str(pid))
        logging.info("Writing PID %s to '%s'", pid, path)
    except:
        raise
    finally:
        try:
            pidfile.close()
        except:
            pass

def remove(path):
    try:
        # make sure we delete our pidfile
        logging.info("Removing pidfile '%s'", path)
        os.unlink(path)
    except:
        pass

I'm going to follow up this post another on how I added a simple concept of "models" and an easy way to perform MySQL transactions. Let me know if you have any specific questions!

Related posts:

1. Python data sharing in the multiprocessing module
2. Tornado 1.0 Released
3. Async DNS Resolution in Tornado’s AsyncHttpClient (curl multi, c-ares)