Pylons Worker Threads

Chris Moos — Wed, 04 Mar 2009 07:03:53 +0000

I was reading a thread over at pylons-discuss about worker threads in pylons. Worker threads are useful if you need to execute a long running task, but want to return to the user immediately. You could run a new thread per request, but if you have many requests, it is probably better to queue things up.

To do this, you start a thread and use python’s Queue for managing the tasks. Here is a very basic implementation:

config/environment.py:

1
2
3
4
5
6
7
8
9
10
11
12
13

# PYLONS IMPORTS

from myapp.lib.myworker import start_myworker

def load_environment(global_conf, app_conf):
"""Configure the Pylons environment via the ``pylons.config``
object
"""

# PYLONS STUFF GOES HERE

# start worker
start_myworker()

lib/myworker.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

import Queue
import threading

worker_q = Queue.Queue()

class MyWorkerThread(threading.Thread):
def run(self):
print 'Worker thread is running.'

while True:
msg = worker_q.get()
try:
# do a long running task...
print 'We got %s, do something with it!' % (msg)
except Exception, e:
print 'Unable to process in worker thread: ' + str(e)
worker_q.task_done()

def start_myworker():
worker = MyWorkerThread()
worker.start()

And in your controller….

1
2
3
4
5

from myapp.lib.myworker import worker_q

class WorkerController(BaseController):
def do_task(self):
worker_q.put({'id': generate_some_unique_id(), 'msg': 'your data goes here'})

chris moos's blog » threads

Pylons Worker Threads