StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>I was trying to find the final word on mmap / read performance on Linux and I came across a nice post (<a href="http://marc.info/?l=linux-kernel&m=95496636207616&w=2" rel="noreferrer">link</a>) on the Linux kernel mailing list. It's from 2000, so there have been many improvements to IO and virtual memory in the kernel since then, but it nicely explains the reason why <code>mmap</code> or <code>read</code> might be faster or slower.</p> <ul> <li>A call to <code>mmap</code> has more overhead than <code>read</code> (just like <code>epoll</code> has more overhead than <code>poll</code>, which has more overhead than <code>read</code>). Changing virtual memory mappings is a quite expensive operation on some processors for the same reasons that switching between different processes is expensive.</li> <li>The IO system can already use the disk cache, so if you read a file, you'll hit the cache or miss it no matter what method you use.</li> </ul> <p>However,</p> <ul> <li>Memory maps are generally faster for random access, especially if your access patterns are sparse and unpredictable.</li> <li>Memory maps allow you to <em>keep</em> using pages from the cache until you are done. This means that if you use a file heavily for a long period of time, then close it and reopen it, the pages will still be cached. With <code>read</code>, your file may have been flushed from the cache ages ago. This does not apply if you use a file and immediately discard it. (If you try to <code>mlock</code> pages just to keep them in cache, you are trying to outsmart the disk cache and this kind of foolery rarely helps system performance).</li> <li>Reading a file directly is very simple and fast.</li> </ul> <p>The discussion of mmap/read reminds me of two other performance discussions:</p> <ul> <li><p>Some Java programmers were shocked to discover that nonblocking I/O is often slower than blocking I/O, which made perfect sense if you know that nonblocking I/O requires making more syscalls.</p></li> <li><p>Some other network programmers were shocked to learn that <code>epoll</code> is often slower than <code>poll</code>, which makes perfect sense if you know that managing <code>epoll</code> requires making more syscalls.</p></li> </ul> <p><strong>Conclusion:</strong> Use memory maps if you access data randomly, keep it around for a long time, or if you know you can share it with other processes (<code>MAP_SHARED</code> isn't very interesting if there is no actual sharing). Read files normally if you access data sequentially or discard it after reading. And if either method makes your program less complex, do <em>that</em>. For many real world cases there's no sure way to show one is faster without testing your actual application and NOT a benchmark.</p> <p>(Sorry for necro'ing this question, but I was looking for an answer and this question kept coming up at the top of Google results.)</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload