Databases
StackOverflow2013
Postsdbo
POUsing lots of MappedByteBuffer's in READ_WRITE mode slows down Windows 7 to a crawl
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUsing lots of MappedByteBuffer's in READ_WRITE mode slows down Windows 7 to a crawl
    primarykey
    data
    text
    <p>I'm trying to use Java MappedByteBuffers in READ_WRITE mode to map a large file (tens of GB) or a set of numerous smallish files (~128MB). This is to implement high performance B-trees.</p> <p>My problem is that on my Windows 7 laptop with 8GB of RAM and JDK 7, everything goes fine until the physical memory is full and the OS starts actually writing the data to the files. At this point, Windows slows down to a crawl. The I/Os seem to completely starve any other activity. The mouse pointer can barely move and I generally end up having to force reboot the machine.</p> <p>The following code demonstrates the issue:</p> <pre><code>public static void testMap() throws Exception { MappedByteBuffer[] mbbs = new MappedByteBuffer[512]; for (int i = 0; i &lt; 512; i++) { System.out.printf("i=%d%n", i); RandomAccessFile raf = new RandomAccessFile(String.format("D:/testMap.%d.bin", i), "rw"); mbbs[i] = raf.getChannel().map(FileChannel.MapMode.READ_WRITE, 0, 128*1024*1024); for (int j = 0; j &lt; 128*1024; j++) { mbbs[i].put(j*1024, (byte)(i*j)); } } } </code></pre> <p>I don't mind if the I/Os take some time. After a while, the OS must actually start to write the bytes into the files anyway. But, here, the process essentially starves the whole OS. How can I avoid this? </p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USallingeek
    1. USalex137
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POUsing lots of MappedByteBuffer's in READ_WRITE mode slows down Windows 7 to a crawl
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POUsing lots of MappedByteBuffer's in READ_WRITE mode slows down Windows 7 to a crawl
      1. USSoftware Mechanic
      1. VTFavorite
    1. COIt is important to recognize that there are going to be resource limitations when working with scarce resources. Is there a reason you need 512 open memory mapped files available at the same time? Depending on your use case there are a number of different approaches you might take including pooling scarce resources (open file handles).
      singulars
      1. POUsing lots of MappedByteBuffer's in READ_WRITE mode slows down Windows 7 to a crawl
      1. USallingeek
    2. COIn reality, I expect the files to be accessed pretty much randomly. That's why I would like to be able to keep all the views mapped at the same time. This seems to be accepted as an OK practice, on 64-bit machine of course. See http://stackoverflow.com/questions/9261316/memory-mapped-mappedbytebuffer-or-direct-bytebuffer-for-db-implementation for example. It does work just fine for reading, but not completely well for writing. I suspect that after a while any memory access from any process forces the OS to swap out some of the dirty page loaded by the JVM process. Any suggestion to avoid this?
      singulars
      1. POUsing lots of MappedByteBuffer's in READ_WRITE mode slows down Windows 7 to a crawl
      1. USalex137
    3. COA pooled solution would work reasonably well depending on how many files you expect to be accessed concurrently. Using a pooled approach you would be able to adjust for performance by varying the pool size. In any case, when the system runs out of RAM the OS will have to start swapping. This is why the whole machine comes to a crawl.
      singulars
      1. POUsing lots of MappedByteBuffer's in READ_WRITE mode slows down Windows 7 to a crawl
      1. USallingeek
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload