Finding and fixing memory leaks in Python

Finding and fixing memory leaks can be a real challenge, but luckily Python has some pretty good tools for spotting these things. I have already written about this issue and this post will go into more details on how to spot and debug a memory leak in MySQLdb, which is the standard Python wrapper for MySQL.

Introducing Dozer

Dozer is a WSGI middleware that displays sparklines of Python object counts and allows you to trace their referents. It can be used with any WSGI application (including Django, which is WSGI compatible).

To install Dozer simply do:

sudo easy_install Dozer

To use Dozer, simply wrap your WSGI application:

from dozer import Dozer
wsgi_app = Dozer(wsgi_app)

To use it from Django, you'll do something like this if you run mod_wsgi:

import os, sys
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'

import django.core.handlers.wsgi
from dozer import Dozer

application = django.core.handlers.wsgi.WSGIHandler()
application = Dozer(application)

Using and understanding Dozer

Dozer shows sparklines of object counts, the non-leaky sparklines look like this:

Normal sparklines

If you have a memory leak, then the sparkline is always increasing!

Memory leak sparkline

If you have the above issue, then something is leaking memory and eventually your application will crash!

To debug a memory leak, simply look at the trace and see which objects are being created without being freed:

Trace the memory leak

Debugging more you'll find that string_decoder is from MySQLdb. Next step is to figure out why it's leaking and fix the leak.

Fixing MySQLdb

The issue was in the C code of MySQL, even thought the way string_decoder is passed is really dirty:

db = proxy(self)
def _get_string_decoder():
    def string_decoder(s):
        return s.decode(string_decoder.charset)
    return string_decoder

Anyway, I was fixing this issue and got the idea of check out MySQLdb 1.2.3c1 - - where luckily for me this issue is solved!

So there it is, how to find and fix memory leaks in Python web applications.

31. Mar 2009 Code · Python · Tips
© Amir Salihefendic