StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Be aware that the address space size for "64bit operating systems" is not necessarily covering the full 64bit range.</p> <p>On x64 (64bit x86, aka 'AMD64'), for example, the actual available virtual address range is "only" 2x128TB, i.e. 48bit in two disjoint 47bit chunks. On some SPARC systems, it's 2x2TB, or 2x8TB (41/44 bits). This is due to the way the MMU works on these platforms.</p> <p>In addition to these architectural limitations, the way the operating system lays out your address spaces also plays a role here.<br> 64bit Windows on x64, for example, limits the virtual address size of an (even 64bit) application to 8TB (each kernel and user side<sup><a href="http://msdn.microsoft.com/en-us/magazine/cc300794.aspx" rel="noreferrer">1</a></sup>).</p> <p>On UN*X systems (including Linux, MacOSX and the *BSDs), there is <code>RLIMIT_AS</code> that one can query for via <code>getrlimit()</code>, and - within system-specific limits - adjust via <code>setrlimit</code>. The <code>ulimit</code> command uses those. They return/set the upper bound, though, i.e. the allowed total virtual address space, including all mappings that a process can create (via <code>mmap()</code> or the <code>malloc</code> backend, <code>sbrk()</code>). But the total address space size is different from the maximum size of a single mapping ...</p> <p>Given this, it's not that hard to exhaust the virtual address space even on 64bit Linux; just try, for a test, to <code>mmap()</code> the same 500GB file, say, two hundred times. The mmap will fail, eventually.</p> <p>In short:</p> <ul> <li><code>mmap()</code> will definitely fail once you're out of virtual address space. Which is, somewhat surprisingly, achievable on many "64bit" architectures simply because virtual addresses might have <em>less than</em> 64 significant bits. The exact cutoff depends on your CPU and your operating system, and an upper bound can be queried via <code>getrlimit(RLIMIT_AS)</code> or set via <code>setrlimit(RLIMIT_AS)</code>. </li> <li>Address space fragmentation can occur on 64bit by frequently using <code>mmap()</code> / <code>munmap()</code> in different orders and with different-sized blocks. This will ultimately limit the maximum size you'll be able to map as a single chunk. Predicting when exactly this will happen is hard since it both depends on your "mapping history" and the operating systems' virtual address space allocation algorithm. If ASLR (address space layout randomization) is done by the OS, it might be unpredictable in detail, and not exactly reproducible.</li> <li><code>malloc()</code> will also fail no later than reaching the total VA limit, on systems (like Linux) where overcommit allows you to "ask" for more memory than there is in the system (physical + swap).</li> <li>On machines / operating systems where no overcommit is enabled, both <code>malloc()</code> and <code>mmap()</code> with <code>MAP_ANON</code> and/or <code>MAP_PRIVATE</code> will fail when physical + swap is exhausted, because these types of mappings require backing store by actual memory or swap.</li> </ul> <p>Little technical update: Like x86 and sparc mentioned above, the new ARMv8 (64bit ARM, called "AArch64" in Linux) MMU also has a "split" address space / an address space hole - out of the 64 bits in an address, only 40 are relevant. Linux gives 39bit for user, virtual addresses <code>0 ...</code> onwards, 39bit for kernel, virtual addresses <code>... 0xFFFFFFFF.FFFFFFFF</code>, so the limit there is 512GB (minus what's in use already at the time the app tries the <code>mmap</code>).<br> See <a href="http://article.gmane.org/gmane.linux.kernel/1324085" rel="noreferrer">comments here</a> (from the AArch64 architecture enabler kernel patch series).</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload