Databases
StackOverflow2013
Postsdbo
POMost efficient/fastest way to get a single file from a directory
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMost efficient/fastest way to get a single file from a directory
    primarykey
    data
    text
    <p>What is the most efficient and fastest way to get a single file from a directory using Python?</p> <p>More details on my specific problem:<br> I have a directory containing a lot of pregenerated files, and I just want to pick a random one. Since I know that there's no really efficient way of picking a random file from a directory other than listing all the files first, my files are generated with an already random name, thus they are already randomly sorted, and I just need to pick the first file from the folder.</p> <p>So my question is: how can I pick the first file from my folder, without having to load the whole list of files from the directory (nor having the OS to do that, my optimal goal would be to force the OS to just return me a single file and then stop!).</p> <p>Note: I have a lot of files in my directory, hence why I would like to avoid listing all the files to just pick one.</p> <p>Note2: each file is only picked once, then deleted to ensure that only new files are picked the next time (thus ensuring some kind of randomness).</p> <p><strong>SOLUTION</strong></p> <p>I finally chose to use an index file that will store:</p> <ul> <li>the index of the current file to be picked (eg: 1 for file1.ext, 2 for file2.ext, etc..)</li> <li>the index of the last file generated (eg: 1999 for file1999.ext)</li> </ul> <p>Of course, this means that my files are not generated with a random name anymore, but using a deterministic incrementable pattern (eg: "file%s.ext" % ID)</p> <p>Thus I have a near constant time for my two main operations:</p> <ul> <li>Accessing the next file in the folder</li> <li>Counting the number of files that are left (so that I can generate new files in a background thread when needed).</li> </ul> <p>This is a specific solution for my problem, for more generic solutions, please read the accepted answer.</p> <p>Also you might be interested into these two other solutions I've found to optimize the access of files and directory walking using Python:</p> <ul> <li><a href="http://blog.fraggod.net/2010/12/oslistdir-and-oswalk-in-python-without-lists-by-the-grace-of-c-api-generator-and-recursion-custom-stack" rel="nofollow">os.walk optimized</a></li> <li><a href="http://python-fam.sourceforge.net/" rel="nofollow">Python FAM (File Alteration Monitor)</a></li> </ul>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USgaborous
    1. USgaborous
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POMost efficient/fastest way to get a single file from a directory
      1. This table or related slice is empty.
      1. VTUpMod
    1. COFigure it out once, then set up your own kind of database to make it quicker?
      singulars
      1. POMost efficient/fastest way to get a single file from a directory
      1. USPearsonArtPhoto
    2. CODirectories are *not* sorted. Rather, they list items in the current directory structure order (on-disk structure), which *could* be the creation order.
      singulars
      1. POMost efficient/fastest way to get a single file from a directory
      1. USMartijn Pieters
    3. COI avoided using DBMS for a reason, because I don't need any indexation of any sort. I don't want to pick a specific file, just any file, and only a single one. @Martijn: what I said was a simplification to better convey my idea. Files are sorted by inode, but that's not the point here: I just want to get the first file because my files are already random so I don't have to care about randomly picking a file: picking just any file in this directory will already give me a random result.
      singulars
      1. POMost efficient/fastest way to get a single file from a directory
      1. USgaborous
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload