Pylons Worker Threads

I was reading a thread over at pylons-discuss about worker threads in pylons. Worker threads are useful if you need to execute a long running task, but want to return to the user immediately. You could run a new thread per request, but if you have many requests, it is probably better to queue things up.

To do this, you start a thread and use python’s Queue for managing the tasks. Here is a very basic implementation:

config/environment.py:

# PYLONS IMPORTS

from myapp.lib.myworker import start_myworker

def load_environment(global_conf, app_conf):
    """Configure the Pylons environment via the ``pylons.config``
    object
    """
    
    # PYLONS STUFF GOES HERE
    
    # start worker
    start_myworker()

lib/myworker.py:

import Queue
import threading

worker_q = Queue.Queue()

class MyWorkerThread(threading.Thread):
    def run(self):
        print 'Worker thread is running.'
        
        while True:
            msg = worker_q.get()
            try:
                # do a long running task...
                print 'We got %s, do something with it!' % (msg)
            except Exception, e:
                print 'Unable to process in worker thread: ' + str(e)
            worker_q.task_done()                
    
def start_myworker():
    worker = MyWorkerThread()
    worker.start()

And in your controller….

from myapp.lib.myworker import worker_q

class WorkerController(BaseController):    
    def do_task(self):
      worker_q.put({'id': generate_some_unique_id(), 'msg': 'your data goes here'})