Databases
StackOverflow2013
Postsdbo
POTraversing files on a distributed file system
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POTraversing files on a distributed file system
    primarykey
    data
    text
    <p>I have a filesystem with a few hundred million files (several petabytes) and I want to get pretty much everything that stat would return and store it in some sort of database. Right now, we have an MPI program that is fed directory names from a central queue and worker nodes that slam NFS (which can handle this without trying too hard) with stat calls. The worker nodes then hit postgres to store the results. </p> <p>Although this works, it's very slow. A single run will take over 24 hours on a modern 30 node cluster.</p> <p>Does anyone have any ideas for splitting up the directory structure instead of having a centralized queue (I'm under the impression that exact algorithms for this are NP hard)? Also, I've been considering replacing postgres with something like MongoDB's autosharding with several routers (since postgres is currently a huge bottleneck).</p> <p>I'm pretty much just looking for ideas in general on how this setup could be improved.</p> <p>Unfortunately, using something like the 2.6 kernel audit subsystem is probably out of the question since it would be extremely difficult (in a political way) to get that running on every machine that hits this filesystem.</p> <p>If it matters, every machine (several thousand) using this filesystem is running linux 2.6.x.</p> <p>The actual primary purpose of this is to find files that are older than a certain date so we can have the ability to delete them. We also want to collect data in general on how the filesystem is being used.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJon Bringhurst
    1. USJon Bringhurst
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POTraversing files on a distributed file system
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhere are your bottlenecks? Database I/O, Disk (i.e. stat) I/O, processing,... ?
      singulars
      1. POTraversing files on a distributed file system
      1. USGaretJax
    2. COAlso, before MongoDB I would consider using Redis. It seems a much more appropriate solution to your use case (finally, it depends on the queries you will do) and is faster than MongoDB.
      singulars
      1. POTraversing files on a distributed file system
      1. USGaretJax
    3. CO@GaretJax Currently the biggest bottlenecks are the stat calls and db writes. I'll take a closer look at redis as well.
      singulars
      1. POTraversing files on a distributed file system
      1. USJon Bringhurst
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload