Databases
StackOverflow2013
Postsdbo
POFastest file access/storage?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POFastest file access/storage?
    primarykey
    data
    text
    <p>I have about 750,000,000 files I need to store on disk. What's more is I need to be able to access these files randomly--any given file at any time--in the <b>shortest time possible</b>. What do I need to do to make accessing these files fastest?</p> <p>Think of it like a hash table, only the hash keys are the filenames and the associated values are the files' data.</p> <p>A coworker said to organize them into directories like this: if I want to store a file named "foobar.txt" and it's stored on the D: drive, put the file in "D:\f\o\o\b\a\r.\t\x\t". He couldn't explain <i>why</i> this was a good idea though. Is there anything to this idea?</p> <p>Any ideas?</p> <p><b>The crux of this is <i>finding</i> a file. What's the fastest way to find a file <i>by name</i> to open?</b></p> <p>EDIT:</p> <p><li>I have no control over the file system upon which this data is stored. It's going to be NTFS or FAT32.</li> <li>Storing the <i>file data</i> in a database is not an option.</li> <li>Files are going to be very small--maximum of probably 1 kb.</li> <li>The drives are going to be solid state.</li> <li>Data access is virtually random, but I could probably figure out a priority for each file based on how often it is requested. Some files will be accessed much more than others.</li> <li>Items will constantly be added, and sometimes deleted.</li> <li>It would be impractical to consolidate multiple files into single files because there's no logical association between files.</li> <li>I would love to gather some metrics by running tests on this stuff, but that endeavour could become as consuming as the project itself!</li></p> <p>EDIT2:</p> <p>I want to upvote several thorough answers, whether they're spot-on or not, and cannot because of my newbie status. Sorry guys!</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJamesBrownIsDead
    1. USJamesBrownIsDead
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POFastest file access/storage?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POFastest file access/storage?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POFastest file access/storage?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIs this data static (750mil is it), or are you adding to it (Adding more files on a periodic basis)? Can it be read only or do you also need to be able to update files? Is it truly random file access, or are there any sort of access patterns you might observe on closer inspection?
      singulars
      1. POFastest file access/storage?
      1. USScanningcrew
    2. COUpdated question to answer this. (More files added on periodic basis, files deleted somewhat infrequently. Access is random, but some files will be accessed much more than others.)
      singulars
      1. POFastest file access/storage?
      1. USJamesBrownIsDead
    3. CORe your EDIT2 comment, you only need 15 rep to vote up. See http://stackoverflow.com/faq for details.
      singulars
      1. POFastest file access/storage?
      1. USGreg Hewgill
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload