Databases
StackOverflow2013
Postsdbo
POquickest way to count the number of files in a directory containing hundreds of thousands of files
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POquickest way to count the number of files in a directory containing hundreds of thousands of files
    primarykey
    data
    text
    <p>In a solaris system that processes large numbers of files and stores their information in a database (yes i know that using the database is the quickest way to get information about the number of files we have). I need fast way to monitor the files as they progress through the system on their way to being stored in a database.</p> <p>Currently I use a perl script that reads in the directory to an array and then grabs the size of the array and sends it to a monitoring script. Unfortunately as our system grows this monitor is getting more and more slow.</p> <p>I am looking for a method that will operate much more quickly instead of pausing and updating every 15-20 seconds after performing the count operation on all the directories involved.</p> <p>I am relatively certain that my bottleneck is the read directory into array operation.</p> <p>I don't need any information about the files, I don't need sizes or file names, just the number of files in the directory.</p> <p>In my code I do not count hidden files or the text files I use to hold configuration information. It would be great if this functionality was preserved but is certainly not mandatory.</p> <p>I have found some references to counting inodes with C code or something along those lines but I am not very experienced in that area.</p> <p>I would like to make this monitor as real-time as possible.</p> <p>The perl code I use looks like this:</p> <pre><code>opendir (DIR, $currentDir) or die "Cannot open directory: $!"; @files = grep ! m/^\./ &amp;&amp; ! /config_file/, readdir DIR; # skip hidden files and config files closedir(DIR); $count = @files; </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USAndrew
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POquickest way to count the number of files in a directory containing hundreds of thousands of files
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POquickest way to count the number of files in a directory containing hundreds of thousands of files
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POquickest way to count the number of files in a directory containing hundreds of thousands of files
      1. This table or related slice is empty.
      1. VTUpMod
    1. COuse perl threads or forks to minimize waiting time
      singulars
      1. POquickest way to count the number of files in a directory containing hundreds of thousands of files
      1. USСухой27
    2. CO@mpapec I like the idea, in the best case scenario this would still make my waiting time close to the amount of time that it would be if i executed this code on only my longest directory? If so, that isn't a bad idea but unfortunately the majority of my directories have less than 50 files while one or two have thousands. I would love to find a way to get away from reading in every file in the directory entirely.
      singulars
      1. POquickest way to count the number of files in a directory containing hundreds of thousands of files
      1. USAndrew
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload