Celery + mysql as results and broker

[Recif Deregle]

Hi all,

I work in a company who is giving a try to celery. So far its seems exactly
what we need
but i have pb configuring it to use a mysql backend for both broker &
results.

i know this is not the privileged way but we already have to integrate with
an existent solution

does an expert can give me a hand to have this working ?

can't find any ref to the error i am having or any tutos for using it with
mysql.

any help much appreciated.

cheers

=== task01.py ===
from celery import Celery

# define a queue
celery = Celery('task01', backend='database',
broker='sqla+mysql://dbuser:***@dbhost/celery')

# load the configuration
celery.config_from_object('celeryconfig')

# a task definition
@celery.task
def add(x, y):
return x + y

=== celeryconfig.py ==
## Broker settings.
BROKER_URL = "sqla+mysql://dbuser:***@dbhost/celery"

## Using the database to store task state and results.
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "sqla+mysql://dbuser:***@dbhost/celery"

# echo enables verbose logging from SQLAlchemy.
# CELERY_RESULT_ENGINE_OPTIONS = {"echo": True}

# timezone
CELERY_TIMEZONE = 'Europe/Paris'

=== running the queue ===
# /opt/celery/bin/celery -A task01 --loglevel=info worker
<snip>
[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] return backend(app=self,
url=url)
[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] File
"/opt/celery/lib/python2.6/site-packages/celery-3.0.11-py2.6.egg/celery/backends/database/__init__.py",
line 70, in __init__
[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] 'Missing connection string!
Do you have '
[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] ImproperlyConfigured:
Missing connection string! Do you have CELERY_RESULT_DBURI set to a real
value?

[Ask Solem]

Post by Recif Deregle
Hi all,
I work in a company who is giving a try to celery. So far its seems exactly what we need
but i have pb configuring it to use a mysql backend for both broker & results.
i know this is not the privileged way but we already have to integrate with an existent solution
does an expert can give me a hand to have this working ?
can't find any ref to the error i am having or any tutos for using it with mysql.
CELERY_RESULT_BACKEND = "database"

The RESULT_DBURI setting is always a SQLAlchemy connection url so shouldn't
have the 'sqla+' prefix:

CELERY_RESULT_DBURI = "mysql://dbuser:***@dbhost/celery"


AFAIR the sqlalchemy broker transport has a serious flaw in that
it never removes any messages. I fixed the Django transport to clean up
messages from time to time, but the sqla transport has not been updated yet.

If you are dedicated to using sqla this shouldn't be too hard to implement,
you can look at the Django transport to see how I did it there:
https://github.com/celery/kombu/blob/master/kombu/transport/django/managers.py#L50-78

Or you can just add a periodic task to delete rows from the Message table with
the visible flag set.

The implementation of both is rather rudimentary since I don't have time
to maintain them, but it shouldn't take much work to make much better.

[Recif Deregle]

Hi Ask,

Thanks for the answer. Indeed that was my last try to make it works. But
the same behavior is hapenning when using :

CELERY_RESULT_DBURI = "mysql://dbuser:***@dbhost/celery"

Do you have any idea of what i'm missing ?

The clear of the messages queue should not be a problem, and if it works
well we might go for RabbitMQ.

Thanks for your help
Cheers

Post by Recif Deregle

Post by Recif Deregle
Hi all,
I work in a company who is giving a try to celery. So far its seems

exactly what we need

Post by Recif Deregle
but i have pb configuring it to use a mysql backend for both broker &

results.

Post by Recif Deregle
i know this is not the privileged way but we already have to integrate

with an existent solution

Post by Recif Deregle
does an expert can give me a hand to have this working ?
can't find any ref to the error i am having or any tutos for using it

with mysql.

Post by Recif Deregle
CELERY_RESULT_BACKEND = "database"

The RESULT_DBURI setting is always a SQLAlchemy connection url so shouldn't
AFAIR the sqlalchemy broker transport has a serious flaw in that
it never removes any messages. I fixed the Django transport to clean up
messages from time to time, but the sqla transport has not been updated yet.
If you are dedicated to using sqla this shouldn't be too hard to implement,
https://github.com/celery/kombu/blob/master/kombu/transport/django/managers.py#L50-78
Or you can just add a periodic task to delete rows from the Message table with
the visible flag set.
The implementation of both is rather rudimentary since I don't have time
to maintain them, but it shouldn't take much work to make much better.
--
Ask Solem
twitter.com/asksol | [phone]

[Recif Deregle]

For the record, it seems i fixed my own problem in 2 steps :

1. back to CELERY_RESULT_DBURI = "mysql://dbuser:***@dbhost/celery"
settings
2. set PYTHONPATH in /etc/default/celery

Cheers

Post by Recif Deregle
Hi Ask,
Thanks for the answer. Indeed that was my last try to make it works. But
Do you have any idea of what i'm missing ?
The clear of the messages queue should not be a problem, and if it works
well we might go for RabbitMQ.
Thanks for your help
Cheers

Post by Recif Deregle

Post by Recif Deregle
Hi all,
I work in a company who is giving a try to celery. So far its seems

exactly what we need

Post by Recif Deregle
but i have pb configuring it to use a mysql backend for both broker &

results.

Post by Recif Deregle
i know this is not the privileged way but we already have to integrate

with an existent solution

Post by Recif Deregle
does an expert can give me a hand to have this working ?
can't find any ref to the error i am having or any tutos for using it

with mysql.

Post by Recif Deregle
CELERY_RESULT_BACKEND = "database"

The RESULT_DBURI setting is always a SQLAlchemy connection url so shouldn't
AFAIR the sqlalchemy broker transport has a serious flaw in that
it never removes any messages. I fixed the Django transport to clean up
messages from time to time, but the sqla transport has not been updated yet.
If you are dedicated to using sqla this shouldn't be too hard to implement,
https://github.com/celery/kombu/blob/master/kombu/transport/django/managers.py#L50-78
Or you can just add a periodic task to delete rows from the Message table with
the visible flag set.
The implementation of both is rather rudimentary since I don't have time
to maintain them, but it shouldn't take much work to make much better.
--
Ask Solem
twitter.com/asksol | [phone]