per-worker initialization to prevent shared SQL connection pools?

[egradman]

My tasks import some SQLAlchemy code that creates an engine. I
believe that the engine is created prior to the worker processes
forking, and that the connection pool shared among processes is
causing some disconnection issues. Is there some way of doing per-
worker initialization, so each worker can instantiate its own engine?

I'm not using SQLAlchemy for any celery functionality (queueing,
tombstones); I'm only using it inside the tasks. I'm using the
postgres dbapi.

[JoNO .]

Did you try the `worker_process_init` signal ?

http://ask.github.com/celery/reference/celery.signals.html#worker-signals

Post by egradman
My tasks import some SQLAlchemy code that creates an engine. I
believe that the engine is created prior to the worker processes
forking, and that the connection pool shared among processes is
causing some disconnection issues. Is there some way of doing per-
worker initialization, so each worker can instantiate its own engine?
I'm not using SQLAlchemy for any celery functionality (queueing,
tombstones); I'm only using it inside the tasks. I'm using the
postgres dbapi.
--
You received this message because you are subscribed to the Google Groups
"celery-users" group.
To unsubscribe from this group, send email to
.
For more options, visit this group at
http://groups.google.com/group/celery-users?hl=en.

[Gijs Koot]

The worker_process_init signal is in principle a good place to initialize a
connection, however, during execution of a task you don't have access to
the worker (except for some far-fetched multiprocessing / multithreading
magic), so how would you be getting hold of the connection? I've seen this
one
http://stackoverflow.com/questions/14526249/celery-worker-database-connection-pooling
but it seems suspect. Thanks for helping.