[celery-users] memory leak using redis for broker and backend

[Chris Withers]

Hi All,

I'm experiencing some serious memory leak problems with workers in a
django/celery combo using redis for the broker and the backend.

The crux of the issue for me, at the moment, is that both heapy and meliae
(injected via pyrasite) show that python knows of about 119Mb, while the
RSS of the worker process is 3.3GB!
How can I find out what's using the remainder of the memory (or at least
what's leaking it!)

We're using:

celery 3.1.11
kombu 3.0.26
billiard 3.3.0.20
py-redis 2.10.3

This appears to have started when we began using redis as the backend for
our task results.

Has anyone else experienced anything like this?

How would I go about debugging what's taking up all the memory that heapy
doesn't know about?

cheers,

Chris

[Chris Withers]

If it helps, celery settings are:

CELERY_TASK_RESULT_EXPIRES=timedelta(hours=6)
CELERY_MAX_TASKS_PER_CHILD=1000

cheers,

Chris

Post by Chris Withers
Hi All,
I'm experiencing some serious memory leak problems with workers in a
django/celery combo using redis for the broker and the backend.
The crux of the issue for me, at the moment, is that both heapy and meliae
(injected via pyrasite) show that python knows of about 119Mb, while the
RSS of the worker process is 3.3GB!
How can I find out what's using the remainder of the memory (or at least
what's leaking it!)
celery 3.1.11
kombu 3.0.26
billiard 3.3.0.20
py-redis 2.10.3
This appears to have started when we began using redis as the backend for
our task results.
Has anyone else experienced anything like this?
How would I go about debugging what's taking up all the memory that heapy
doesn't know about?
cheers,
Chris