Databases
StackOverflow2013
Postsdbo
POStore huge std::map, mostly on disk
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POStore huge std::map, mostly on disk
    primarykey
    data
    text
    <p>I've got a C++ program that's likely to generate a HUGE amount of data -- billions of binary records of varying sizes, most probably less than 256 bytes but a few stretching to several K. Most of the records will seldom be looked at by the program after they're created, but some will be accessed and modified regularly. There's no way to tell which are which when they're created.</p> <p>Considering the volume of data, there's no way I can store it all in memory. But as the data only needs to be indexed and accessed by its number (a 64-bit integer), I <em>don't</em> want the overhead of a full-fledged database program. Ideally I'd like to treat it as an <code>std::map</code> with its data stored on disk until requested.</p> <p>Is there an already-written library that will do what I'm looking for, or do I need to write it myself?</p> <p>EDIT: After some thought, I realized that Rob Walker's answer had a valid point: I'd be hard-pressed to get anywhere near the same kind of data integrity out of a home-brew class that I'd get from a real database.</p> <p>Although BerkeleyDB (as suggested by RHM) looks like it would do exactly what we're looking for, the dual-licensing is a headache that we don't want to deal with. When we're done with the code and can prove that it would benefit noticeably from BerkeleyDB (which it probably would), we'll reexamine the issue.</p> <p>I did look at Ferruccio's suggestion of stxxl, but I wasn't able to tell how it would handle the program being interrupted and restarted (maybe with changes). With that much data, I'd hate to just scrap what it had already completed and start over every time, if some of the data could be saved.</p> <p>So we've decided to use an SQLite database, at least for the initial development. Thanks to everyone who answered or voted.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USHead Geek
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POStore huge std::map, mostly on disk
      1. USBrian R. Bondy
      1. VTFavorite
    2. VO
      singulars
      1. POStore huge std::map, mostly on disk
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POStore huge std::map, mostly on disk
      1. USEvilTeach
      1. VTFavorite
    1. COI am curious. what domain is this for? Why/what is the data?
      singulars
      1. POStore huge std::map, mostly on disk
      1. USTim
    2. COSorry, I'm not at liberty to say anything about it just yet.
      singulars
      1. POStore huge std::map, mostly on disk
      1. USHead Geek
    3. COOK. I am guessing some kind of analog media or parts of photos. Good luck
      singulars
      1. POStore huge std::map, mostly on disk
      1. USTim
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload