StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>@James Wahlin, thanks a lot for helping me out.</p> <p>As it turned out the main cause of latency was mongodb <code>GLOBAL LOCK</code> itself. We had a lock percentage which was averaging at 5% and sometime peaks up to 30-50% and that results in the slow queries.</p> <p>If you are facing this issue, the first thing you have to do it enable mongodb MMS service (<code>mms.10gen.com</code>), which will give you a lot of insights on what exactly is happening in your db server.</p> <p>In our case the LOCK PERCENTAGE was really high and there were multiple reasons for it. First thing you have to do to figure it out is to read mongodb documentation on concurrency, <a href="http://docs.mongodb.org/manual/faq/concurrency/" rel="nofollow">http://docs.mongodb.org/manual/faq/concurrency/</a></p> <p>The reason for lock can be in application level, mongodb or hardware.</p> <p>1) Our app was doing a lot of <code>updates</code> and each update(more than <code>100 ops/sec</code>) holds a <code>global lock</code> in mongodb. The issue was that when an update happens for an entry which is not in memory, mongo will have to load the data into memory first and then update (in memory) and the whole process happens while inside the <code>global lock</code>. If say the whole thing takes 1 sec to complete (<code>0.75sec</code> to load the data from disk and <code>0.25sec</code> to update in memory), the whole rest of the update calls waits (for the entire <code>1 sec</code>) and such updates starts queuing up... and you will notice more and more slows requests in your app server.</p> <p>The solution for it (while it might sound silly) is to <code>query</code> for the same data before you make an <code>update</code>. What it effectively does is moving the 'load data to memory' (0.75sec) part out of the <code>global lock</code> which greatly reduces your <code>lock percentage</code></p> <p>2) The other main cause of global lock is mongodb's data <code>flush</code> to disk. Basically in every 60sec (or less) mongodb (or the OS) writes the data to disk and a <code>global lock</code> is held during this process. (This kinda explains the random slow queries). In your MMS stats, see the graph for <code>background flush avg</code>... if its high, that means you have to get faster disks.</p> <p>In our case, we moved to a new <code>EBS optimized</code> instance in <code>EC2</code> and also bumped our provisioned <code>IOPS</code> from <code>100</code> to <code>500</code> which almost halved the <code>background flush avg</code> and the servers are much happier now.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload