I learned some rather important facts about cURL's multi interface (which makes it possible to perform asynchronous HTTP requests and what Python's Tornado framework uses under the hood in it's AsyncHttpClient helper).

I was investigating some intermittent issues in an application at work - transient DNS issues were causing the application to become unresponsive. This was confusing at first because it was written to perform HTTP requests asynchronously. The important thing here is that it was specifically DNS resolution that was failing. As I dug deeper I realized that simple health checks, ones that did not perform any HTTP requests, were also hanging... something had to be blocking.

Taking a look at the Tornado source, unsurprisingly, it was leveraging the multi functionality of libcurl (via pycurl). What was surprising to me was that the DNS resolution portion of the multi interface, by default, blocks on non-windows installations. Only when compiled with c-ares support does it perform these async.

You learn something new every day!

Related posts:

1. Python’s Tornado has swept me off my feet
2. Tornado 1.0 Released
3. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger