Databases
StackOverflow2013
Postsdbo
POHow to Guarantee Message delivery with Celery?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to Guarantee Message delivery with Celery?
    primarykey
    data
    text
    <p>I have a python application where I want to start doing more work in the background so that it will scale better as it gets busier. In the past I have used Celery for doing normal background tasks, and this has worked well. </p> <p>The only difference between this application and the others I have done in the past is that I need to guarantee that these messages are processed, they can't be lost. </p> <p>For this application I'm not too concerned about speed for my message queue, I need reliability and durability first and formost. To be safe I want to have two queue servers, both in different data centers in case something goes wrong, one a backup of the other.</p> <p>Looking at Celery it looks like it supports a bunch of different backends, some with more features then the others. The two most popular look like redis and RabbitMQ so I took some time to examine them further.</p> <p><strong>RabbitMQ:</strong> Supports durable queues and clustering, but the problem with the way they have clustering today is that if you lose a node in the cluster, all messages in that node are unavailable until you bring that node back online. It doesn't replicated the messages between the different nodes in the cluster, it just replicates the metadata about the message, and then it goes back to the originating node to get the message, if the node isn't running, you are S.O.L. Not ideal. </p> <p>The way they recommend to get around this is to setup a second server and replicate the file system using DRBD, and then running something like pacemaker to switch the clients to the backup server when it needs too. This seems pretty complicated, not sure if there is a better way. Anyone know of a better way?</p> <p><strong>Redis:</strong> Supports a read slave and this would allow me to have a backup in case of emergencies but it doesn't support master-master setup, and I'm not sure if it handles active failover between master and slave. It doesn't have the same features as RabbitMQ, but looks much easier to setup and maintain.</p> <p><strong>Questions:</strong></p> <ol> <li><p>What is the best way to setup celery so that it will guarantee message processing.</p></li> <li><p>Has anyone done this before? If so, would be mind sharing what you did?</p></li> </ol>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USKen Cochrane
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to Guarantee Message delivery with Celery?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to Guarantee Message delivery with Celery?
      1. USTed Naleid
      1. VTFavorite
    3. VO
      singulars
      1. POHow to Guarantee Message delivery with Celery?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COas for the rabbitmq failover, I'm hearing rumours that something simpler will be available soon!
      singulars
      1. POHow to Guarantee Message delivery with Celery?
      1. USasksol
    2. CORedis can be durable if you set the append_only setting. But redis still doesn't support message acknowledgements, which means that a message is redelivered if the worker doesn't ack it. Celery redis support emulates this, but only as well as is possible to do on the client side, which means that any unacked message may be lost if the worker is killed abruptly or there is a power failure. See http://ask.github.com/celery/faq.html#should-i-use-retry-or-acks-late
      singulars
      1. POHow to Guarantee Message delivery with Celery?
      1. USasksol
    3. COMaybe you can get away with not losing messages if you set CELERY_DISABLE_RATE_LIMITS=True, set CELERYD_PREFETCH_MULTIPLIER=1, set CELERY_ACKS_LATE=True, and run with the solo pool. But would have to verify that.
      singulars
      1. POHow to Guarantee Message delivery with Celery?
      1. USasksol
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload