Databases
StackOverflow2013
Postsdbo
POUsing NIO, do I need to care about R/W on block-boundaries?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUsing NIO, do I need to care about R/W on block-boundaries?
    primarykey
    data
    text
    <p><strong>Background</strong></p> <p>A lot of work has gone into optimizing database design, especially in the realm of the most optimal ways to read and write data from disks (both spindle and SSD).</p> <p>The knowledge that has come out of the work suggests that reading and writing on block boundaries, matching the block sizes of the filesystem you are running on, is the most optimal approach.</p> <p><strong>Question</strong></p> <p>Say I am operating in a relatively low-memory environment and want to use a small 32MB memory-mapped file to read and write the contents of a huge 500GB file.</p> <p>If I were using Java's NIO mechanisms, specifically the <a href="http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html" rel="nofollow">MappedByteBuffer</a> (Java's memory-mapped file mechanism), would I need to take care to execute READ and WRITE operations on block boundaries (e.g. 4KB) into memory before pairing out the data I needed, or can I just issue R/W ops at any location I want and allow the operating system, VM paging logic, filesystem and storage firmware handle the optimization of the operations and culling of additional block data I didn't need as-needed?</p> <p><strong>Additional Detail</strong></p> <p>The reason for the question is in database design, I see this <em>obsessive</em> focus on block-optimization to the point that there doesn't seem to exist a world where you would ever just read and write data without the concept of a block.</p> <p>What confuses me is that the filesystem is the one enforcing the block units of operation, why would my higher level app need to worry about this then? If I want the 17,631 bytes at offset 71, can't I just grab them and read them in, or is it really faster for me to figure out that the read operation starts at block 0 and falls across the boundaries of blocks 0, 1 and 2... read all of those 3 blocks in to an internal byte[], then cull out the 17,631 bytes I wanted in the first place?</p> <p>If the literature on DB design wasn't so religious about this block idea, the question would have never come up in my mind, but because it is, I am wondering if I am missing a critical detail here WRT filesystems and optimal block device I/O.</p> <p>Thank you for reading.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USRiyad Kalla
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POUsing NIO, do I need to care about R/W on block-boundaries?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POUsing NIO, do I need to care about R/W on block-boundaries?
      1. USmtsz
      1. VTFavorite
    3. VO
      singulars
      1. POUsing NIO, do I need to care about R/W on block-boundaries?
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload