[celery-users] All Django DB Connections are Persistent

[Kevin Chevalier]

Hello everyone,

Please let me know if I misunderstand anything. I tried to research this
as much as I could before posting.

It is my understanding that Django DB connections should reset at the
beginning of each
task. https://github.com/celery/celery/blob/master/celery/fixups/django.py#L217

However, when I run a task with concurrency, this signal handler is never
called, therefore the connection is never closed. This means that the
Django DB connection is reused from task to task.

The DjangoWorkerFixup registers for the signals.task_prerun signal once,
when the WorkController is created in the main worker process.
https://github.com/celery/celery/blob/master/celery/fixups/django.py#L184

However, when the task is run, a different signal is checked for
receivers. https://github.com/celery/celery/blob/master/celery/app/trace.py#L343

These signals had different ids, I think because signals.py was imported
again in the child process.



I dug in some more, and noticed that DjangoWorkerFixup.install is never
called in the child process because the signals.worker_init signal is never
called in the child process.

So in the main process with the WorkController, the handler is registered,
but task_prerun signal is never triggered. In the child process, the
handler is never registered, but the task_prerun signal tries to trigger.
This handler never runs when a task is run.


The result of this is that if one task loses a connection, all subsequent
tasks fail until Celery is restarted. Does anyone know if I have something
misconfigured, or if I am misunderstanding anything?

Thanks,

Kevin

[Kevin Chevalier]

I've been looking into this more. It appears that it is only an issue with
CELERYD_FORCE_EXECV = True. We have this enabled because of this bug.
https://github.com/celery/celery/issues/572

I've have a minimal configuration that displays this problem, if anyone is
interested.

Post by Kevin Chevalier
Hello everyone,
Please let me know if I misunderstand anything. I tried to research this
as much as I could before posting.
It is my understanding that Django DB connections should reset at the
beginning of each task.
https://github.com/celery/celery/blob/master/celery/fixups/django.py#L217
However, when I run a task with concurrency, this signal handler is never
called, therefore the connection is never closed. This means that the
Django DB connection is reused from task to task.
The DjangoWorkerFixup registers for the signals.task_prerun signal once,
when the WorkController is created in the main worker process.
https://github.com/celery/celery/blob/master/celery/fixups/django.py#L184
However, when the task is run, a different signal is checked for
receivers.
https://github.com/celery/celery/blob/master/celery/app/trace.py#L343
These signals had different ids, I think because signals.py was imported
again in the child process.
I dug in some more, and noticed that DjangoWorkerFixup.install is never
called in the child process because the signals.worker_init signal is never
called in the child process.
So in the main process with the WorkController, the handler is registered,
but task_prerun signal is never triggered. In the child process, the
handler is never registered, but the task_prerun signal tries to trigger.
This handler never runs when a task is run.
The result of this is that if one task loses a connection, all subsequent
tasks fail until Celery is restarted. Does anyone know if I have something
misconfigured, or if I am misunderstanding anything?
Thanks,
Kevin