hacking and gonzo since 2000

Parallel processing made easy in Python

A while ago I wrote about @run_in_thread decorator that can be used to run a function in a thread. Here is an extension that can be used to run multiple functions in threads. This is especially useful if you are scripting something and you need parallel processing.

How to use it:

import os
from functools import partial

def expensive_operation(sleep_time):
    print 'Sleeping for %s seconds...' % sleep_time
    os.popen('sleep %s' % (sleep_time))
    print 'Slept for %s seconds' % sleep_time

run_in_threads(
    partial(expensive_operation, 10),
    partial(expensive_operation, 5),
    partial(expensive_operation, 15)
)

How it's implemented:

import threading

def run_in_threads(*functions):
    threads = []

    for fn in functions:
        thread = threading.Thread(target=fn)
        thread.start()
        threads.append(thread)

    for thread in threads:
        thread.join()

Code · Python · Tips • 31. Mar 2010