I was reading a thread over at pylons-discuss about worker threads in pylons. Worker threads are useful if you need to execute a long running task, but want to return to the user immediately. You could run a new thread per request, but if you have many requests, it is probably better to queue things up.
To do this, you start a thread and use python’s Queue for managing the tasks. Here is a very basic implementation:
config/environment.py:
# PYLONS IMPORTS from myapp.lib.myworker import start_myworker def load_environment(global_conf, app_conf): """Configure the Pylons environment via the ``pylons.config`` object """ # PYLONS STUFF GOES HERE # start worker start_myworker()
lib/myworker.py:
import Queue import threading worker_q = Queue.Queue() class MyWorkerThread(threading.Thread): def run(self): print 'Worker thread is running.' while True: msg = worker_q.get() try: # do a long running task... print 'We got %s, do something with it!' % (msg) except Exception, e: print 'Unable to process in worker thread: ' + str(e) worker_q.task_done() def start_myworker(): worker = MyWorkerThread() worker.start()
And in your controller….
from myapp.lib.myworker import worker_q class WorkerController(BaseController): def do_task(self): worker_q.put({'id': generate_some_unique_id(), 'msg': 'your data goes here'})
Thanks Chris,
You’ve done three things for me here that I am thankful for, one is show me how to call helper programs from the lib directory, two is a simple example of threading in python, and three is using the built in python queue.
Much appreciated, and very clear too.
Thanks a lot,
Chris.
Glad you found it helpful. Also, if you want to pass a SQLAlchemy object into the queue, you will experience thread issues. If you wanted to pass a user into your queue, either pass the individual values you need (user.name, user.username) etc,., or pass (user.id) and then in the thread you query and get the object there.
Thanks for the great code! I am having problems with passing variables to the thread though. I keep getting:
No object (name: app_globals) has been registered for this thread
In the start_worker, I have:
worker = TagMail(app_globals.MAX_TAG_LENGTH)
worker.start()
Any help?
Thanks
Maybe you need to import app_globals? Pylons right?
I have imported it. Still no luck. Oh well. Thanks!
from pylons import app_globals
x = app_globals.MAX_TAG_LENGTH
worker = TagMail(x)
worker.start()
@Anil simply change start_myworker() to start_myworker(max_tag_len) and call it in load_environment with appropriate variable. app_globals is a pylons global which is available for pylons threads (the threads pylons creates to handle requests).