Also, this project is being hosted on GitHub @ http://github.com/mreiferson/py-wkhtmltox.

wkhtmltox

What is wkhtmltox you ask? It's a utility built on Nokia's Qt framework for converting HTML (including images, CSS, and Javascript) to a PDF or image. When Qt introduced it's webkit module it made it relatively easy to leverage it's rendering engine to produce high quality output.

wkhtmltox 0.10.0beta5 introduced libwkhtmltox - a simple C API making it possible to embed this functionality in higher-level scripting languages. I jumped at the opportunity to build Python bindings...

First I tried SIP, which is actually used for the Qt bindings. A variety of issues coupled with poor documentation led me to search for other solutions. I won't bore you with the details because it seems there are better ways...

Cython

Note: I'm not talking about CPython (the default Python implementation written in C). I'm talking about a toolset to build C extensions for Python. The two major use cases being speed or, in my case, wrapping a C library.

It's really easy to get up and running, Cython the language is a hybrid of C and Python. You write scripts with a .pyx extension, make things modular with .pxd files, and handle all the building and installation with distutils.

Let's take a look at the API libwkhtmltox exposes (I'm only showing lines relevant to this post):

struct wkhtmltopdf_global_settings;
typedef struct wkhtmltopdf_global_settings wkhtmltopdf_global_settings;

struct wkhtmltopdf_object_settings;
typedef struct wkhtmltopdf_object_settings wkhtmltopdf_object_settings;

struct wkhtmltopdf_converter;
typedef struct wkhtmltopdf_converter wkhtmltopdf_converter;

CAPI int wkhtmltopdf_init(int use_graphics);
CAPI int wkhtmltopdf_deinit();

CAPI const char * wkhtmltopdf_version();

CAPI wkhtmltopdf_global_settings * wkhtmltopdf_create_global_settings();
CAPI wkhtmltopdf_object_settings * wkhtmltopdf_create_object_settings();

CAPI int wkhtmltopdf_set_global_setting(wkhtmltopdf_global_settings * settings, const char * name, const char * value);
CAPI int wkhtmltopdf_set_object_setting(wkhtmltopdf_object_settings * settings, const char * name, const char * value);

CAPI wkhtmltopdf_converter * wkhtmltopdf_create_converter(wkhtmltopdf_global_settings * settings);
CAPI void wkhtmltopdf_destroy_converter(wkhtmltopdf_converter * converter);

CAPI int wkhtmltopdf_convert(wkhtmltopdf_converter * converter);
CAPI void wkhtmltopdf_add_object(wkhtmltopdf_converter * converter, wkhtmltopdf_object_settings * setting, const char * data);

CAPI int wkhtmltopdf_http_error_code(wkhtmltopdf_converter * converter);

And let's also look at the basic example provided with the wkhtmltopdf source distribution (again, only relevant lines shown):

wkhtmltopdf_init(false);
gs = wkhtmltopdf_create_global_settings();
wkhtmltopdf_set_global_setting(gs, "out", "test.pdf");
os = wkhtmltopdf_create_object_settings();
wkhtmltopdf_set_object_setting(os, "page", "http://doc.trolltech.com/4.6/qstring.html");
c = wkhtmltopdf_create_converter(gs);
wkhtmltopdf_add_object(c, os, NULL);
wkhtmltopdf_convert(c);
wkhtmltopdf_destroy_converter(c);
wkhtmltopdf_deinit();

I found it helpful to try to identify which methods needed to be exposed to user-space and which ones could be safely abstracted behind a cleanly wrapped Pythonic API. I wrote the following test script to work towards:

import wkhtmltox

pdf = wkhtmltox.Pdf()
pdf.set_global_setting('out', 'test.pdf')
pdf.set_object_setting('path', 'http://www.google.com')
pdf.convert()

The initialization of a Pdf instance handles all of the internal libwkhtmltox initialization. See below for the .pyx script. Of interest is how C structs and functions are exposed to the Cython script and then wrapped, with appropriate names, as Python methods of a class. Astute observers will also notice that some of the function declarations differ slightly from the original libwkhtmltox header file. In most cases Cython doesn't need the const, CAPI and other specific declarations. Also, bint hints that even though the return value is an int we should convert this to a boolean on the Python side.

cdef extern from "wkhtmltox/pdf.h":
    struct wkhtmltopdf_converter:
        pass

    struct wkhtmltopdf_object_settings:
        pass

    struct wkhtmltopdf_global_settings:
        pass

    bint wkhtmltopdf_init(int use_graphics)
    bint wkhtmltopdf_deinit()
    char *wkhtmltopdf_version()

    wkhtmltopdf_global_settings *wkhtmltopdf_create_global_settings()
    wkhtmltopdf_object_settings *wkhtmltopdf_create_object_settings()

    bint wkhtmltopdf_set_global_setting(wkhtmltopdf_global_settings *settings, char *name, char *value)
    bint wkhtmltopdf_get_global_setting(wkhtmltopdf_global_settings *settings, char *name, char *value, int vs)
    bint wkhtmltopdf_set_object_setting(wkhtmltopdf_object_settings *settings, char *name, char *value)
    bint wkhtmltopdf_get_object_setting(wkhtmltopdf_object_settings *settings, char *name, char *value, int vs)

    wkhtmltopdf_converter *wkhtmltopdf_create_converter(wkhtmltopdf_global_settings *settings)
    void wkhtmltopdf_destroy_converter(wkhtmltopdf_converter *converter)

    bint wkhtmltopdf_convert(wkhtmltopdf_converter *converter)
    void wkhtmltopdf_add_object(wkhtmltopdf_converter *converter, wkhtmltopdf_object_settings *setting, char *data)

    int wkhtmltopdf_http_error_code(wkhtmltopdf_converter *converter)

cdef extern from "wkhtmltox/image.h":
    struct wkhtmltoimage_global_settings:
        pass

    struct wkhtmltoimage_converter:
        pass

    bint wkhtmltoimage_init(int use_graphics)
    bint wkhtmltoimage_deinit()
    char *wkhtmltoimage_version()

    wkhtmltoimage_global_settings *wkhtmltoimage_create_global_settings()

    bint wkhtmltoimage_set_global_setting(wkhtmltoimage_global_settings *settings, char *name, char *value)
    bint wkhtmltoimage_get_global_setting(wkhtmltoimage_global_settings *settings, char *name, char *value, int vs)

    wkhtmltoimage_converter *wkhtmltoimage_create_converter(wkhtmltoimage_global_settings *settings, char *data)
    void wkhtmltoimage_destroy_converter(wkhtmltoimage_converter *converter)

    bint wkhtmltoimage_convert(wkhtmltoimage_converter *converter)

    int wkhtmltoimage_http_error_code(wkhtmltoimage_converter *converter)

cdef class Pdf:
    cdef wkhtmltopdf_global_settings *_c_global_settings
    cdef wkhtmltopdf_object_settings *_c_object_settings
    cdef bint last_http_error_code

    def __cinit__(self):
        wkhtmltopdf_init(0)
        self._c_global_settings = wkhtmltopdf_create_global_settings()
        self._c_object_settings = wkhtmltopdf_create_object_settings()

    def __dealloc__(self):
        wkhtmltopdf_deinit();

    def set_global_setting(self, char *name, char *value):
        return wkhtmltopdf_set_global_setting(self._c_global_settings, name, value)

    def set_object_setting(self, char *name, char *value):
        return wkhtmltopdf_set_object_setting(self._c_object_settings, name, value)

    def convert(self):
        cdef wkhtmltopdf_converter *c
        c = wkhtmltopdf_create_converter(self._c_global_settings)
        wkhtmltopdf_add_object(c, self._c_object_settings, NULL)
        ret = wkhtmltopdf_convert(c)
        self.last_http_error_code = wkhtmltopdf_http_error_code(c)
        wkhtmltopdf_destroy_converter(c)
        return ret

    def http_error_code(self):
        return self.last_http_error_code

cdef class Image:
    cdef wkhtmltoimage_global_settings *_c_global_settings
    cdef bint last_http_error_code

    def __cinit__(self):
        wkhtmltoimage_init(0)
        self._c_global_settings = wkhtmltoimage_create_global_settings()

    def __dealloc__(self):
        wkhtmltoimage_deinit();

    def set_global_setting(self, char *name, char *value):
        return wkhtmltoimage_set_global_setting(self._c_global_settings, name, value)

    def convert(self):
        cdef wkhtmltoimage_converter *c
        c = wkhtmltoimage_create_converter(self._c_global_settings, NULL)
        ret = wkhtmltoimage_convert(c)
        self.last_http_error_code = wkhtmltoimage_http_error_code(c)
        wkhtmltoimage_destroy_converter(c)
        return ret

    def http_error_code(self):
        return self.last_http_error_code

This is really my first attempt to get something working. I'm sure there are bugs and perhaps better ways to go about this. I always welcome questions/feedback.

I'm going to continue to support this project at http://github.com/mreiferson/py-wkhtmltox - watch it!

Related posts:

1. Python data sharing in the multiprocessing module
2. Django URL Parameter Passing and Python Strings
3. Django Up In Your CRON

At this time, Matt was admittedly more of a programmer than I was.  While our web "programming" skills were roughly equal, Matt was also into C/C++ programming.  He wrote an Asteroids-like game to teach his brother math, and he would dissect gaming engines like the one used in Doom.  At one point we even thought of developing our own first person shooter based on our neighborhood and school, the premise being, we had to rescue everyone from zombies and aliens.  Again, youthful ambition, and not a bit of naivety.  Sometime in 1995 or 1996 we registered our first domain name, perplexed.com, with a $100 investment from my grandmother.  The name "perplexed" represented everything about us at the time: young and curious, but sometimes confused about the state of the world we were in.  Why did people do certain (illogical) things?  What were we going to do with our lives when we got to college?  Will the Knicks ever beat the Bulls in the playoffs?  It is my contention that perplexed.com is the greatest domain name ever registered, and I cannot articulate how angry we are with ourselves that we didn't renew it. More on that later

We decided perplexed.com would host two different sites: Matt's Game Programming MegaSite, and my VB Programming MegaSite.  This arrangement allowed us both to build sites related to our interests.  We tried to keep the layouts the same so we'd have a consistent design across the domain; two frames, the left being the menu, the right being the content.  The color scheme was black background with white text.  We tried to make all our images using PhotoShop.  They came out ok in a cheesy way, as you'd expect from two programmers.  GPMega would focus on all aspects of game programming, especially engine design and DirectX.  Matt had a ton of C/C++ code up, as well as tutorials, links, software, and even a MIDI player so you can listen to a MIDI version of Van Halen's "Panama" while you browsed.  VBMega was concerned with Visual Basic programming, and the main focus was on programs that you could use to manipulate earlier versions of AOL.  For example, posting ASCII art in chat rooms, knocking people offline, pinging the service every few minutes so you wouldn't get knocked off, as well as other more nefarious features.  I ran this portion of the site, which allowed me to learn VB programming while at the same time piss off my friends by booting them off AOL over IM.  Everyone was a winner!

We made money the old-fashioned way, by selling ad space.  We signed up for various banner exchanges before settling with 24/7.  They had a different name back then, I just can't remember what it was.  In any case, the sites both grew very quickly, and the checks were arriving.  The first few months we made $150, then $200, then $300.  At the high point we were pulling in $1500 a month, strictly through ads.  When you're 15 years old in high school and you're walking around with that kind of money in your pocket, you feel on top of the world.  While the rest of our classmates ate the cafeteria food, we walked over to the deli across the street and treated ourselves to bacon egg and cheese sandwiches.  Going to the mall was fun: your girl wants some lunch?  No problem.  New shirt?  Done.  New programming book?  Buy two!  One month, perplexed.com accounted for a little more than 1% of the total ad impressions for every site that advertised with 24/7.  We were experiencing impressive growth.  Perhaps the most stunning thing to happen to us during this time was one day when we were in CompUSA.  Matt was leafing through a book, Tricks Of the Windows Game Programming Gurus, when he saw GPMega mentioned!  We didn't even know the author had referenced us.  It was shocking; we were open-mouthed.

Unfortunately, all good things must come to an end.  In retrospect, the way Perplexed ended is sad indeed, and the only way I can explain it is stupidity.  In October of 1998, my family and I moved to another town, which was only 15 minutes away from where Matt lived, but to two kids with no cars who would now be going to different schools, it might as well have been a million miles.  We kept in touch and continued to hang out, but it was hard to run the sites at the time.  So many of our ideas came from lunchtime brainstorming, playing ball after school, or Friday night coding sessions.  Now that we weren't seeing each other on a day to day basis, the importance of the sites and the partnership that created them began to fall.  It sounds insane in this age of telecommuting, but back then we didn't know any better.  And priorities changed as we got older.  When you get a new car and you're hanging out with your girlfriend, and doing whatever high school kids do, who wants to be up at 3am on a Friday writing HTML and answering emails?  The nail in the coffin was the failure to renew the domain name.  To this day I don't know why we didn't do it.  I can't explain it; it was a stupid decision and one of the biggest regrets of my youth.

At this point, Matt and I are seniors in different high schools, preparing to attend different colleges.  We're still great friends but the lack of face time has meant that our priorities diverged a bit.  It's worth mentioning that my grades were better in high school and college after I moved away.  I failed out of Advanced Math in seventh grade, but got A's in Calc I, II, and III in college.  Go figure!  In all seriousness, the fact that Perplexed.com is now part of an ad farm is a huge source of pain and regret.  But all is not lost.  Stay tuned for Part III, where I will bring the story up to the present time, and offer my reflections on the business and pleasure of web programming.

Related posts:

1. 13 Years On The Web – A Retrospective, Part I
2. Ad Revenue as a Business Model – Eat Dinner First

Related posts:

1. PHP Drinks Java
2. Python’s Tornado has swept me off my feet
3. Deployment Using Capistrano / Webistrano via Rails / Phusion Passenger