Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>If your original data is saved in HBase, you can design your key so rows will be sorted by time. You can then scan the table with a start row defined as the last row you scanned yesterday + 1. You can also have the key start with the day. As rows are sorted by key, you can easily start with the first row of a desired day and stop at the next day.</p> <p>If you create your rows as:</p> <pre><code>long currentTimeMS = System.currentTimeMillis(); long currentDay = currentTimeMS / (1000 * 60 * 60 * 60 * 24); Put put = new Put(Bytes.add(Bytes.toBytes(currentDay), "some other key stuff".getBytes())); // add columns... hbaseTable.put(put); </code></pre> <p>You can scan a day's worth of data with:</p> <pre><code>long currentDay = currentTimeMS / (1000 * 60 * 60 * 60 * 24); long yesterday = currentDay - 1; Scan dayScan = new Scan(); dayScan.setStartRow(Bytes.toBytes(yesterday)); dayScan.setStopRow(Bytes.toBytes(currentDay)); // create map reduce job with dayScan </code></pre> <p>There are some libraries like <a href="http://joda-time.sourceforge.net/" rel="nofollow">Joda Time</a> that make time calculations easier and the code more readable.</p> <p>You can also try <a href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setTimeRange%28long,%20long%29" rel="nofollow">scan.setTimeRange()</a> for a similar outcome. But that assumes you insert and never update the source rows, as it actually operates on the <em>udpate</em> time of column versions. It might also be slower as the data might not be close together thanks to sorting by row key. Overall, this doesn't seem like the recommended way to go. But for quick and dirty prototyping, it works.</p> <p>If you are scanning data straight from HDFS, then you can achieve something similar by simply saving data to a different directory every day. You can then only scan yesterday's directory and nothing else.</p>
    singulars
    1. This table or related slice is empty.
    1. POIncremental update to MapReduce results
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USkichik
    1. USkichik
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POIncremental update to MapReduce results
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COwhile we can definitely read new data with your method from a MapReduce job.....How do we actually merge the results with the previously computed results? This was my actual question....I apologize if this wasn't clear
      singulars
      1. PO
      1. USrahul
    2. COThat really depends on what your results are. If they are really big, you may need a second MapReduce job to combine them. If they're small, you can just update and save the result somewhere. More information on the actual data would help figure out the best solution.
      singulars
      1. PO
      1. USkichik
    3. COhaving a second mapreduce job might actually work in my case where I am generating Inverted Indexes for search.....this new job can merge the indexes
      singulars
      1. PO
      1. USrahul
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload