Databases
StackOverflow2013
Postsdbo
POOut of memory w/ mapdb
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOut of memory w/ mapdb
    text
    copied!<p>I'm getting an out of memory error when testing mapdb. Given that the whole idea of the project is to serialize data structures to disk and avoid memory problems, I figure I'm doing something wrong. Any ideas what I'm doing wrong? Or is there a bug?</p> <pre><code> @Test public void testLarge() throws Exception { final HTreeMap&lt;UUID, String&gt; storage = DBMaker.newTempHashMap(); String string = createDataSize(250); ArrayList&lt;UUID&gt; keys = new ArrayList&lt;&gt;(); for (int i = 0; i &lt; 320000; i++) { final UUID key = UUID.randomUUID(); storage.put(key, string); keys.add(key); } for (UUID key : keys) { assertNotNull(storage.get(key)); } for (UUID key : keys) { storage.remove(key); } assertEquals("nothing left", 0, storage.size()); } /** * Creates a message of size @msgSize in KB. */ private static String createDataSize(int msgSize) { // Java chars are 2 bytes msgSize = msgSize / 2; msgSize = msgSize * 1024; StringBuilder sb = new StringBuilder(msgSize); for (int i = 0; i &lt; msgSize; i++) { sb.append('a'); } return sb.toString(); } } </code></pre> <p>Stack trace. Line 29 in my function corresponds to the "assertNotNull(storage.get(key));" line.</p> <pre><code>java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2367) at java.lang.String.&lt;init&gt;(String.java:168) at org.mapdb.SerializerBase.deserializeString(SerializerBase.java:724) at org.mapdb.SerializerBase.deserialize(SerializerBase.java:932) at org.mapdb.SerializerBase.deserialize(SerializerBase.java:731) at org.mapdb.HTreeMap$1.deserialize(HTreeMap.java:134) at org.mapdb.HTreeMap$1.deserialize(HTreeMap.java:123) at org.mapdb.StorageDirect.recordGet2(StorageDirect.java:536) at org.mapdb.StorageDirect.get(StorageDirect.java:201) at org.mapdb.EngineWrapper.get(EngineWrapper.java:50) at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:163) at org.mapdb.EngineWrapper.get(EngineWrapper.java:50) at org.mapdb.CacheHashTable.get(CacheHashTable.java:85) at org.mapdb.HTreeMap.get(HTreeMap.java:387) at com.sample.StorageTest.testLarge(StorageTest.java:29) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:292) </code></pre> <p>Based on unholysampler's suggestion, I made the following change which fixed the problem.</p> <pre><code>diff -r 35918e46551a src/test/java/com/sample/StorageTest.java --- a/src/test/java/com/sample/StorageTest.java Thu Mar 21 13:40:16 2013 -0600 +++ b/src/test/java/com/sample/StorageTest.java Thu Mar 21 13:42:24 2013 -0600 @@ -16,7 +16,9 @@ @Test public void testLarge() throws Exception { - final HTreeMap&lt;UUID, String&gt; storage = DBMaker.newTempHashMap(); + File tmpFile = File.createTempFile("largeTest", null); + DB db = DBMaker.newFileDB(tmpFile).deleteFilesAfterClose().journalDisable().make(); + final HTreeMap&lt;UUID, String&gt; storage = db.getHashMap("name"); String string = createDataSize(250); @@ -25,6 +27,9 @@ final UUID key = UUID.randomUUID(); storage.put(key, string); keys.add(key); + if (i%100==0) { + db.commit(); + } } for (UUID key : keys) { </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload