StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>I'd like to add some further notes in my response. All other non-Microsoft operating systems we have tested do not suffer from this problem. Linux, FreeBSD, and Mac OS X (this final one on different hardware) all degrade much more gracefully in terms of aggregate bandwidth when moving from one thread to two. Linux for example degraded from ~45 MiB/sec to ~42 MiB/sec. These other operating systems must be reading larger chunks of the file between each seek, and therefor not spending nearly all their time waiting on the disk to seek.</p> <p>Our solution for Windows is to pass the <code>FILE_FLAG_NO_BUFFERING</code> flag to <code>CreateFile</code> and use large (~16MiB) reads in each call to <code>ReadFile</code>. This is suboptimal for several reasons:</p> <ul> <li>Files don't get cached when read like this, so there are none of the advantages that caching normally gives.</li> <li>The constraints when working with this flag are much more complicated than normal reading (alignment of read buffers to page boundaries, etc).</li> </ul> <p>(As a final remark. Does this explain why swapping under Windows is so hellish? Ie, Windows is incapable of doing IO to multiple files concurrently with any efficiency, so while swapping all other IO operations are forced to be disproportionately slow.)</p> <hr> <p><em>Edit to add some further details for Will Dean:</em></p> <p>Of course across these different hardware configurations the raw figures did change (sometimes substantially). The problem however is the consistent degradation in performance that only Windows suffers when moving from one thread to two. Here is a summary of the machines tested:</p> <ul> <li>Several Dell workstations (Intel Xeon) of various ages running Windows 2000, Windows XP (32-bit), and Windows XP (64-bit) with single drive.</li> <li>A Dell 1U server (Intel Xeon) running Windows Server 2003 (64-bit) with RAID 1+0.</li> <li>An HP workstation (AMD Opteron) with Windows XP (64-bit), and Windows Server 2003, and hardware RAID 5.</li> <li>My home unbranded PC (AMD Athlon64) running Windows XP (32-bit), FreeBSD (64-bit), and Linux (64-bit) with single drive.</li> <li>My home MacBook (Intel Core1) running Mac OS X, single SATA drive.</li> <li>My home <a href="http://koolu.com/Koolu-WE-Appliance/Works-Everywhere-Appliance.html" rel="nofollow noreferrer">Koolu</a> PC running Linux. Vastly underpowered compared to the other systems but I demonstrated that even this machine can outperform a Windows server with RAID5 when doing multi-threaded disk reads.</li> </ul> <p>CPU usage on all of these systems was very low during the tests and anti-virus was disabled.</p> <p>I forgot to mention before but we also tried the normal Win32 <code>CreateFile</code> API with the <code>FILE_FLAG_SEQUENTIAL_SCAN</code> flag set. This flag didn't fix the problem.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload