StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PODiagnosing appengine java backend hangs in deployed apps
text
Body
copied!<p>We are using backends to parallelize large batches of work for our users. This involves adding 100's of tasks to a dedicated queue which spreads the tasks across 5 dynamic backends. Many of these tasks do a url fetch to external web services. </p> <p>When we run this for one user it seems to work fine but as soon as we run it for many users (e.g. in a nightly batch run) we see most of the backends eventually stopping processing new tasks. </p> <p>We've tried rate limiting it but that has not helped either. I can manually stop the backends that are frozen and new ones will fire up and start processing until they too freeze. Eventually it all completes after a few iterations of stopping frozen servers. </p> <p>Obviously this is severely limiting our ability to scale so I'm wondering how to diagnose this problem. We cannot reproduce this on localhost since its not really multi-threaded and doesn't truly replicate the deployed data patterns. </p> <p>We could possibly handle this by using pull queues but that's more code to write and I'd rather let the queue/backend scheduler do that work for us. </p> <p>What I'd like to do is attach a profiler or jconsole to a backend - that would immediately tell me where the freezes are coming from. Is that even possible? If not, does anybody have any other tricks that we could use to diagnose frozen backends? </p> <p>fyi : the "frozen" instances have varied levels of memory from 150mb to 250mb i.e. it could be a memory leak but the memory numbers don't really support that, they consume zero cpu when frozen and process no new tasks. </p> <p>Thanks for your attention and help, Steve</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload