Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>It really depends on the specific case but I can think that a possible solution would be to store events in pages and keep in memory just the page directory:</p> <pre><code>struct Page { int id; int timestamp0, timestamp1; int bytes_used; }; </code></pre> <p>Every page only has events for a specific ID and all pages are of the same size (e.g. 4K). When you receive an event you add it to the specific page if it fits, otherwise allocate a new page for that event ID.</p> <p>When doing the searches you can quickly decide by looking at the index which pages from your data file are worth processing and you don't have to process the whole file.</p> <p>Pseudocode for adding an event is:</p> <ol> <li>find last page for ID x</li> <li>if the event doesn't fit in the page allocate a new fresh page</li> <li>store the event and update the index entry for the page</li> </ol> <p>for doing a search you:</p> <ol> <li>for each entry in the index</li> <li>if the entry is about an ID you don't care about then go to next one</li> <li>if <code>(page.timestamp0 &gt;= tsmax || page.timestamp1 &lt; tsmin)</code> then the page doesn't contain any interesting event, go to next one</li> <li>this page contains at least a relevant event; load the page and process the events that are contained in the period <code>tsmin ... tsmax</code> you are interested in.</li> </ol> <p>You can also avoid storing the index in the file (making the event logging operation faster) if you add an ID field once per page. Just when starting the server a full scan of all the data will be needed... may be or not this is an interesting option.</p> <p>You can also easily decide how much data to keep... when no more free pages are available you can reuse (may be after storing away a zipped copy for archival) all pages that only contain events older than a certain date.</p>
    singulars
    1. This table or related slice is empty.
    1. POOptimal data structure for time and source dependent log data for fast browsing?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. US6502
    1. US6502
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POOptimal data structure for time and source dependent log data for fast browsing?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThat's an very interesting thought! Did I understand that correctly? You are primarily sorting the data by ID, each ID forms a bucket / page and get's filled by timestamp/data pairs. If it's full, a new bucket / page for that ID will be generated - and so on. Thus searching will be very fast for the IDs and then a binary search for the timestamp through the filtered dataset.
      singulars
      1. PO
      1. USChris
    2. COThe idea is to keep an index in memory for all pages (using only the shown structure for each page). When you need to do a search you can just scan the index and decide which pages to load and search by filtering on ID and skipping pages that don't intersect your timestamp interval). In each page you only have events for a specific ID (so the ID doesn't need to be stored in the event... you need just timestamp, type and data) and it's not a problem to have events of variable size (you simply go to next page once a page is full and next event doesn't fit).
      singulars
      1. PO
      1. US6502
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload