[celery-users] Unresponsive workers when prefork pool is fully utilized

[Adam Klein]

I'm using Celery in most likely an unconventional way: as the basis for a
compute grid. Each worker has a pre-fork pool equal in size to its number
of cores. Each task pulled off the queue induces a worker process to spawn
a new python process (via subprocess.Popen in a new thread that consumes
the subprocess textual output), and awaits this subprocess's exit via a
thread join. These are long-running, computationally intensive processes
(think numpy/scipy stuff).

The problem is, when a worker's pre-fork pool is fully utilized, the worker
no longer responds to polling via Flower or via inspect() from
celery.task.control. On this worker's log, you will also see info-level
messages such as "missed heartbeat from ***@hostname" where hostname are
the other worker machines. Also, "substantial drift from ***@hostname
may mean clocks are out of sync".

However, this is *not* a problem when I only use up to (n-1) of the n
pre-forked processes. It seems that the worker's n'th pool process, which
isn't blocked, can maintain communication with the rest of the grid, and
keep the worker from going "dark".

Are there any thoughts around how to work around this situation?

I'm using RabbitMQ and CELERYD_PREFETCH_MULTIPLIER = 1 and also locally
controlling how many tasks I send onto the work queue as a best effort
right now, but essentially this requires cooperative grid usage among users.

Thanks!

[Dmitry Malinovsky]

Hello, Adam.

Try using -Ofair option.

Post by Adam Klein
I'm using Celery in most likely an unconventional way: as the basis for a
compute grid. Each worker has a pre-fork pool equal in size to its number
of cores. Each task pulled off the queue induces a worker process to spawn
a new python process (via subprocess.Popen in a new thread that consumes
the subprocess textual output), and awaits this subprocess's exit via a
thread join. These are long-running, computationally intensive processes
(think numpy/scipy stuff).
The problem is, when a worker's pre-fork pool is fully utilized, the
worker no longer responds to polling via Flower or via inspect() from
celery.task.control. On this worker's log, you will also see info-level
may mean clocks are out of sync".
However, this is *not* a problem when I only use up to (n-1) of the n
pre-forked processes. It seems that the worker's n'th pool process, which
isn't blocked, can maintain communication with the rest of the grid, and
keep the worker from going "dark".
Are there any thoughts around how to work around this situation?
I'm using RabbitMQ and CELERYD_PREFETCH_MULTIPLIER = 1 and also locally
controlling how many tasks I send onto the work queue as a best effort
right now, but essentially this requires cooperative grid usage among users.
Thanks!

[Adam Klein]

No help, unfortunately. Same behavior.

Post by Dmitry Malinovsky
Hello, Adam.
Try using -Ofair option.

Post by Adam Klein
I'm using Celery in most likely an unconventional way: as the basis for a
compute grid. Each worker has a pre-fork pool equal in size to its number
of cores. Each task pulled off the queue induces a worker process to spawn
a new python process (via subprocess.Popen in a new thread that consumes
the subprocess textual output), and awaits this subprocess's exit via a
thread join. These are long-running, computationally intensive processes
(think numpy/scipy stuff).
The problem is, when a worker's pre-fork pool is fully utilized, the
worker no longer responds to polling via Flower or via inspect() from
celery.task.control. On this worker's log, you will also see info-level
may mean clocks are out of sync".
However, this is *not* a problem when I only use up to (n-1) of the n
pre-forked processes. It seems that the worker's n'th pool process, which
isn't blocked, can maintain communication with the rest of the grid, and
keep the worker from going "dark".
Are there any thoughts around how to work around this situation?
I'm using RabbitMQ and CELERYD_PREFETCH_MULTIPLIER = 1 and also locally
controlling how many tasks I send onto the work queue as a best effort
right now, but essentially this requires cooperative grid usage among users.
Thanks!

[Adam Klein]

I just tried adding

CELERY_ACKS_LATE = True

This did the trick! Can someone explain to me in simple terms what this
does? I found the docs a bit cryptic.

I suppose I needed this in conjunction with my other setting, CELERYD_PREFETCH_MULTIPLIER
= 1

Post by Adam Klein
No help, unfortunately. Same behavior.

Post by Dmitry Malinovsky
Hello, Adam.
Try using -Ofair option.

Post by Adam Klein
I'm using Celery in most likely an unconventional way: as the basis for
a compute grid. Each worker has a pre-fork pool equal in size to its number
of cores. Each task pulled off the queue induces a worker process to spawn
a new python process (via subprocess.Popen in a new thread that consumes
the subprocess textual output), and awaits this subprocess's exit via a
thread join. These are long-running, computationally intensive processes
(think numpy/scipy stuff).
The problem is, when a worker's pre-fork pool is fully utilized, the
worker no longer responds to polling via Flower or via inspect() from
celery.task.control. On this worker's log, you will also see info-level
may mean clocks are out of sync".
However, this is *not* a problem when I only use up to (n-1) of the n
pre-forked processes. It seems that the worker's n'th pool process, which
isn't blocked, can maintain communication with the rest of the grid, and
keep the worker from going "dark".
Are there any thoughts around how to work around this situation?
I'm using RabbitMQ and CELERYD_PREFETCH_MULTIPLIER = 1 and also locally
controlling how many tasks I send onto the work queue as a best effort
right now, but essentially this requires cooperative grid usage among users.
Thanks!

[Ask Solem]

Post by Adam Klein
I just tried adding
CELERY_ACKS_LATE = True
This did the trick! Can someone explain to me in simple terms what this does? I found the docs a bit cryptic.
I suppose I needed this in conjunction with my other setting, CELERYD_PREFETCH_MULTIPLIER = 1

Some parts of the worker is still operating synchronously:

- Message acknowledgements
- Updating broker QOS (basic.qos) in response to eta tasks or —autoscale
- Publishing results when the process executing the task terminates (WorkerLostError) or if the task is revoked.
- Sending task error emails.
- The actual handlers for the remote control commands (most are optimized to return quickly, but users can also
add custom commands)

So any of these can potentially block the remote control command from being handled in a timely manner.
I have been working on fixing the first three, but this is a huge project that will take some time at this pace.

I have run extensive tests and rarely see any of these block the worker locally, but if the broker is slow
for some reason this can obviously happen.

You can send the USR1 signal to the parent process* to force it to log a traceback of what it’s doing, if you do this a
few times when it’s responsive you’re likely to get some idea of what it’s blocking on.

You can also increase the timeout for the remote control commands: celery inspect —timeout=5
but I would say the best solution is to optimize the workers so that they can handle them in less than a second.


* `pip install setproctitle` and restart worker to see the MainProcess name in ps listings
--
Ask Solem
twitter.com/asksol | [phone]