Finding and fixing memory leaks in Python
Finding and fixing memory leaks can be a real challenge, but luckily Python has some pretty good tools for spotting these things. I have already written about this issue and this post will go into more details on how to spot and debug a memory leak in MySQLdb, which is the standard Python wrapper for MySQL.
Dozer is a WSGI middleware that displays sparklines of Python object counts and allows you to trace their referents. It can be used with any WSGI application (including Django, which is WSGI compatible).
To install Dozer simply do:
sudo easy_install Dozer
To use Dozer, simply wrap your WSGI application:
from dozer import Dozer wsgi_app = Dozer(wsgi_app)
To use it from Django, you'll do something like this if you run mod_wsgi:
import os, sys sys.path.append('/usr/local/django') os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings' import django.core.handlers.wsgi from dozer import Dozer application = django.core.handlers.wsgi.WSGIHandler() application = Dozer(application)
Using and understanding Dozer
Dozer shows sparklines of object counts, the non-leaky sparklines look like this:
If you have a memory leak, then the sparkline is always increasing!
If you have the above issue, then something is leaking memory and eventually your application will crash!
To debug a memory leak, simply look at the trace and see which objects are being created without being freed:
Debugging more you'll find that string_decoder is from MySQLdb. Next step is to figure out why it's leaking and fix the leak.
The issue was in the C code of MySQL, even thought the way string_decoder is passed is really dirty:
db = proxy(self) def _get_string_decoder(): def string_decoder(s): return s.decode(string_decoder.charset) return string_decoder
Anyway, I was fixing this issue and got the idea of check out MySQLdb 1.2.3c1 - - where luckily for me this issue is solved!
So there it is, how to find and fix memory leaks in Python web applications.