Databases
StackOverflow2013
Postsdbo
POTweaking celery for high performance
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POTweaking celery for high performance
    primarykey
    data
    text
    <p>I'm trying to send ~400 HTTP GET requests and collect the results. I'm running from django. My solution was to use celery with gevent.</p> <p>To start the celery tasks I call <strong>get_reports</strong> :</p> <pre><code>def get_reports(self, clients, *args, **kw): sub_tasks = [] for client in clients: s = self.get_report_task.s(self, client, *args, **kw).set(queue='io_bound') sub_tasks.append(s) res = celery.group(*sub_tasks)() reports = res.get(timeout=30, interval=0.001) return reports @celery.task def get_report_task(self, client, *args, **kw): report = send_http_request(...) return report </code></pre> <p>I use 4 workers:</p> <pre><code>manage celery worker -P gevent --concurrency=100 -n a0 -Q io_bound manage celery worker -P gevent --concurrency=100 -n a1 -Q io_bound manage celery worker -P gevent --concurrency=100 -n a2 -Q io_bound manage celery worker -P gevent --concurrency=100 -n a3 -Q io_bound </code></pre> <p>And I use RabbitMq as the broker.</p> <p>And although it works much faster than running the requests sequentially (400 requests took ~23 seconds), I noticed that most of that time was overhead from celery itself, i.e. if I changed <strong>get_report_task</strong> like this:</p> <pre><code>@celery.task def get_report_task(self, client, *args, **kw): return [] </code></pre> <p>this whole operation took ~19 seconds. That means that <strong>I spend 19 seconds only on sending all the tasks to celery and getting the results back</strong></p> <p>The queuing rate of messages to rabbit mq is seems to be bound to 28 messages / sec and I think that this is my bottleneck.</p> <p>I'm running on a win 8 machine if that matters. </p> <p>some of the things I've tried:</p> <ul> <li>using redis as broker</li> <li>using redis as results backend </li> <li><p>tweaking with those settings</p> <p>BROKER_POOL_LIMIT = 500</p> <p>CELERYD_PREFETCH_MULTIPLIER = 0</p> <p>CELERYD_MAX_TASKS_PER_CHILD = 100</p> <p>CELERY_ACKS_LATE = False</p> <p>CELERY_DISABLE_RATE_LIMITS = True</p></li> </ul> <p>I'm looking for any suggestions that will help speed things up.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USlev
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POTweaking celery for high performance
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHey! 28 messages a sec is very low as a publishing rate. I've seen celery+librabbitmq do 100.000 tasks/second using non-persistent messages on my desktop PC. Do you really only measure sending the message, or do you also measure getting the result back? The Redis reusult backend is *not optimized for RPC*. There is a new RPC result backend in the development version (to be Celery 3.1) that is better for this.
      singulars
      1. POTweaking celery for high performance
      1. USasksol
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload