PitStop

StopFinder has a suite of unit and integration tests that probably has just about as much code as the application itself. The first thing I do after the deployment is run the tests. This is in the nature of a smoke test, to make sure that I haven’t messed up syncing with third-party libraries and such. If the tests pass on my local machine, I don’t really expect them to fail when they’re deployed to another host.

While it’s great to run the integration tests on the same database configuration that the app itself will be running on, sometimes your host environment will make that impossible. I deploy Django sites in a lot of different places. Often times, the hosts that I’m working with aren’t sitting in my apartment under my direct control, but are rather owned by folks who make their living by maintaining app environments for people like me.

The Django unittest framework operates by creating a test database at the beginning of every run, and then wiping the tables after each test. At the end of the entire run, the database is dropped. This can cause some problems if you don’t have control of your host. For example, some shared hosting plans will create a single database for you, and disallow you from issuing CREATE DATABASE or DROP DATABASE queries yourself. (My WebFaction account operates like this, for example.)

If you’re in the hosting scenario I just described, you can’t run your tests without hand-editing your database properties, or using a different testing-specific settings file, or something else cludgy like this. This annoyed me when I first encountered it, but I was pretty swamped at the time. I budgeted about 10 minutes to resolving the problem, and came up with this very simple solution.

The idea is to wrap django-admin.py runtests so that I can select the database configuration of my choice. If I don’t have permission to create and drop databases, I’ll use SQLite as the test database. Otherwise, I’ll use the resident database server itself.

Here’s how to do that in two easy steps.

1. Write a script to drive django-admin.py runtests that sets an environment variable when you want to use SQLite (which I stupidly called a “test database” flag). The script then issues a shell call to django-admin:

import sys

_TESTING_COMMAND = 'django-admin.py test'

def run(args):
    """
    Just runs the tests, with the caveat that we've reset the environment
    variable to something we want. If you want to use test database settings,
    specify -tdb.
    """

    if len(args) == 1 and args[0] == "-tdb":
            os.environ["TESTING_DJANGO"] = "THIS_VALUE_DOESN'T_MATTER"

    error_code = os.system(_TESTING_COMMAND)

    if error_code != 0:
        print "Testing command failed: errcode %d" % error_code

if __name__ == "__main__":
    run(sys.argv[1:])

2. In your local settings files, or wherever else you define your database settings, you can override the production values by doing something like:

if "TESTING_DJANGO" in os.environ:
    DATABASE_ENGINE = 'sqlite3'
    DATABASE_NAME = ':memory:'
    DATABASE_USER = ''
    DATABASE_PASSWORD = ''
    DATABASE_HOST = ''
    DATABASE_PORT = ''
    TEST_DATABASE_NAME = ":memory:"

Alternately, you could have the SQLite settings be the default, and then in the local settings file you could decide to bind the database-related properties to the production database settings only if the TESTING_DJANGO flag is not set. Whatever works best for you; this is just the general idea.

Yes, this is sort of hacky, but I had it up and running in a handful of moments and it does frequently come in handy. I was hoping that by running the tests on an in-memory database it would also speed things up a bit, but I guess the table creation and data-loading time overshadows the actual time spent exercising the database in test code; running the StopFinder tests under SQLite actually takes just as long or longer than when I run it against postgres.

Hope this helps someone out there! For all I know, more recent Django builds have this sort of flexibility directly baked-in.

I’m a big fan of Jon Bentley, the author of Programming Pearls. In this book, he discusses elegant solutions to problems that have irritated programmers for years. (I was in Europe for 6 weeks last year, and this was the one book that remained consistently in my bag throughout the whole trip. I’m not joking.)

In chapter 3 of the second edition, Bentley demonstrates how “form letters” (another name for templates) can be used to solve some particularly tedious programming problems.

Like most web frameworks, Django (everyone’s favorite web stack) ships with its own template engine. In pretty much any Django tutorial, you’ll find a section on how to use templates to implement the dynamic pages on your website. This can lead to a sort of tunnel-vision that causes you to forget that templates exist outside of the world of dynamically generated HTML.

However, as Bentley was so kind to remind me, templates are good for so much more than generating HTML pages to be served over HTTP. I often find myself attacking ugly problems with a silly custom solution until I have that aha! moment in which I realize that templates are the perfect answer.

A couple of examples:

Email templates:

I do contract web development here and there. About a year ago, I was working on a client site that sent automated plain-text emails to users in response to certain events. The site owner wanted an easy way to edit these emails. I won’t tell you how silly my first solution to this problem was.

Luckily, I was quick to realize that plain text can be templated just as easily as HTML can. So, I made a template directory specifically for these emails, and documented the variables that would be passed into each template in comments at the top of each relevant file. The code to send the emails involves querying the relevant information from the database, producing a dictionary of key-value pairs from this data, and feeding this into the appropriate template to produce the email text.

The best part was that when the site owner decided that he also wanted to maintain HTML versions of his newsletters, I was able to use template inheritance to keep all of the text for both plaintext and HTML versions of each newsletter in one place.

XML API responses:

My masters thesis was in data management. If you had walked by my desk on any given day when I was working on my thesis project, you probably would have heard me muttering “I hate XML” as I stared maniacally into my editor window.

The StopFinder XML API was one of the last features to be implemented before our 1.0 release. I was starting to get a wee bit tired. Coffee was becoming more of a dependency than a performance enhancer. You know how it is. The last thing I wanted to do was mess around with code to build XML responses. Parsing it is bad enough.

After about five minutes of reading about different Python modules that can build XML documents programmatically, I had the predictable aha! moment. Why not just use a template to specify how the response should look? My favorite thing about using templates to implement XML responses is that the document structure is very obvious to maintenance programmers, in comparison to some nasty module that uses the DOM to build a document in code.

One word of warning with this approach: The natural way to implement an XML API from here is to write a vanilla view function to accept POST requests, or something like this. However, if this view function is in the same app as the rest of your site’s functionality, an out-of-control API consumer could potentially bog down the performance of your entire site. A better thing to do would probably be to run a seperate Django instance specifically for your API. Of course, if the API hits a shared database really hard, you’re probably going to be in trouble no matter what. But that’s a story for another article.

So, the lesson of the day is to remember that templates are a general purpose tool. This is probably obvious to most clever folks, but the slow ones like me occasionally need to be reminded.

As we’ve proudly stated in our site footer, StopFinder is built on a host of technologies that we think are really awesome. These include Python, Django, MochiKit, postgresql, Google Maps, and probably some more I’ve already forgotten about.

However, we weren’t completely familiar with all of these technologies when we started out. We learned quite a few lessons along the way. And now, we’d like to share these lessons with the interested folks in the audience.

So, while we’re working on the next set of StopFinder features, we’re going to take some time out now and again to post technical articles that we think will be of interest to the people out there who are building their own web apps. In fact, I’m working on the first one right now.

Stay tuned! We hope you enjoy them!