Databases
StackOverflow2013
Postsdbo
PObenchmarks: does python have a faster way of walking a network folder?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PObenchmarks: does python have a faster way of walking a network folder?
    primarykey
    data
    text
    <p>I need to walk through a folder with approximately ten thousand files. My old vbscript is very slow in handling this. Since I've started using Ruby and Python since then, I made a benchmark between the three scripting languages to see which would be the best fit for this job.</p> <p>The results of the tests below on a subset of 4500 files on a shared network are</p> <pre><code>Python: 106 seconds Ruby: 5 seconds Vbscript: 124 seconds </code></pre> <p>That Vbscript would be slowest was no surprise but I can't explain the difference between Ruby and Python. Is my test for Python not optimal? Is there a faster way to do this in Python?</p> <p>The test for thumbs.db is just for the test, in reality there are more tests to do.</p> <p>I needed something that checks every file on the path and doesn't produce too much output to not disturb the timing. The results are a bit different each run but not by much.</p> <pre><code>#python2.7.0 import os def recurse(path): for (path, dirs, files) in os.walk(path): for file in files: if file.lower() == "thumbs.db": print (path+'/'+file) if __name__ == '__main__': import timeit path = '//server/share/folder/' print(timeit.timeit('recurse("'+path+'")', setup="from __main__ import recurse", number=1)) </code></pre> <pre class="lang-vb prettyprint-override"><code>'vbscript5.7 set oFso = CreateObject("Scripting.FileSystemObject") const path = "\\server\share\folder" start = Timer myLCfilename="thumbs.db" sub recurse(folder) for each file in folder.Files if lCase(file.name) = myLCfilename then wscript.echo file end if next for each subfolder in folder.SubFolders call Recurse(subfolder) next end Sub set folder = oFso.getFolder(path) recurse(folder) wscript.echo Timer-start </code></pre> <pre><code>#ruby1.9.3 require 'benchmark' def recursive(path, bench) bench.report(path) do Dir["#{path}/**/**"].each{|file| puts file if File.basename(file).downcase == "thumbs.db"} end end path = '//server/share/folder/' Benchmark.bm {|bench| recursive(path, bench)} </code></pre> <p>EDIT: since i suspected the print caused a delay i tested the scripts with printing all 4500 files and also printing none, the difference remains, R:5 P:107 in the first case and R:4.5 P:107 in the latter</p> <p>EDIT2: based on the answers and comments here a Python version that in some cases could run faster by skipping folders</p> <pre><code>import os def recurse(path): for (path, dirs, files) in os.walk(path): for file in files: if file.lower() == "thumbs.db": print (path+'/'+file) def recurse2(path): for (path, dirs, files) in os.walk(path): for dir in dirs: if dir in ('comics'): dirs.remove(dir) for file in files: if file.lower() == "thumbs.db": print (path+'/'+file) if __name__ == '__main__': import timeit path = 'f:/' print(timeit.timeit('recurse("'+path+'")', setup="from __main__ import recurse", number=1)) #6.20102692 print(timeit.timeit('recurse2("'+path+'")', setup="from __main__ import recurse2", number=1)) #2.73848228 #ruby 5.7 </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USpeter
    1. USpeter
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PObenchmarks: does python have a faster way of walking a network folder?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PObenchmarks: does python have a faster way of walking a network folder?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PObenchmarks: does python have a faster way of walking a network folder?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COBloody good question, this. I can't see any obvious improvements that can be made to the Python code. I'm not near a decent internet connection, and so can't download Ruby and test this at the moment.
      singulars
      1. PObenchmarks: does python have a faster way of walking a network folder?
      1. USChinmay Kanchi
    2. COHmmm... having said that, I wonder if the `for` loop is the culprit here. Try `fnames = [name for name in files if name.lower()=='thumbs.db']`, followed by `if fnames: print fnames[0]`.
      singulars
      1. PObenchmarks: does python have a faster way of walking a network folder?
      1. USChinmay Kanchi
    3. COAre you sure that this behaviour is not connected to some network cache policy? The speed differences between Python and Ruby shouldn't be so massive...
      singulars
      1. PObenchmarks: does python have a faster way of walking a network folder?
      1. USPaolo Moretti
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload