Databases
StackOverflow2013
Postsdbo
POJava On-Memory Efficient Key-Value Store
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POJava On-Memory Efficient Key-Value Store
    primarykey
    data
    text
    <p>I have store 111 million key-value pairs (one key can have multiple values - maximum 2/3) whose key are 50 bit Integers and values are 32 bit (maximum) Integers. Now, my requirements are:</p> <blockquote> <ol> <li>Fast Insertion of (Key, Value) pair [allowing duplicates]</li> <li>Fast retrieving of value/values based on key.</li> </ol> </blockquote> <p>A nice solution of it is given <a href="https://stackoverflow.com/questions/10056135/java-hash-multi-map-key-with-multiple-values-implementation">here</a> based on MultiMap. However, I want to store more key-values pairs in main memory with no/little bit performance penalty. I studied from web articles that B+ Tree, R+ Tree, B Tree, Compact Multimap etc. can be a nice solution for that. Can anybody help me:</p> <p>Is there any Java library which satisfies my all those needs properly (above mentioned/other ds also acceptable. no issue with that) ? Actually, I want an efficient java library data structure to store/retrieve key-value/values pairs which takes less memory footprint and must be built in-memory. </p> <p>NB: I have tried with HashMultiMap (Guava with some modification with trove) as mentioned by Louis Wasserman, Kyoto/Tokyo Cabinet etc etc.My experience is not good with disk-baked solutions. So please avoid that :). Another point is that, for choosing library/ds one important point is: keys are 50 bit (so if we assign 64bit) 14 bit will lost and values are 32 bit Int (maximum)- mostly they are 10-12-14 bits. So, we can save space there also. </p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USArpssss
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POJava On-Memory Efficient Key-Value Store
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POJava On-Memory Efficient Key-Value Store
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POJava On-Memory Efficient Key-Value Store
      1. USError_404
      1. VTFavorite
    1. COMost of the space will be lost to support fast insertion and removal. The the more work your insert/removal is the more compact you can be. It seems you should be able to store this in a few GB easily. What is your memory requirement?
      singulars
      1. POJava On-Memory Efficient Key-Value Store
      1. USPeter Lawrey
    2. CO@PeterLawrey, I want to store all 110 million key-value in 5/6 GB.
      singulars
      1. POJava On-Memory Efficient Key-Value Store
      1. USArpssss
    3. COSo if you have 12 bytes for the key and value you can still have about 75% overhead.
      singulars
      1. POJava On-Memory Efficient Key-Value Store
      1. USPeter Lawrey
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload