StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>I'm running a queue system that is dealing with millions of messages per day. Mostly via Amazon SQS, but I'm also running a new Beanstalkd system with over 600,000 msgs in there right now.</p> <p>As is described in <a href="http://www.phpscaling.com/2009/06/23/doing-the-work-elsewhere-sidebar-running-the-worker/" rel="nofollow">a blogpost on the subject</a>, I have shell scripts running in a loop processing messages (a loop within the PHP script to run multiple jobs before returning is also somewhat useful, at least for smaller jobs).</p> <p>Those shell scripts are started with <a href="http://supervisord.org/" rel="nofollow">Supervisord</a>. There's another <a href="http://phpadvent.org/2009/daemonize-your-php-by-sean-coates" rel="nofollow">blog post on the use of that</a> as well. I'm currently running over 800 worker scripts (for a few different types of jobs) across nine machines, all pulling from various queues and putting data back into other queues, writing to the DB or files. Increasing the number of workers per machine is a matter of increasing the "numprocs" (or having it large enough already), and then starting more as required. You could also have say 5 auto-start, and then another block of 50 that are ready to start as required.</p> <p>I find each worker only takes around 20mb of non-shared memory (the rest is common between processes). This does depend on the tasks that the workers do of course. Resizing images can take a lot of effort. It's partly for this reason I have setup to be able to frequently restart the PHP script.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload