Databases
StackOverflow2013
Postsdbo
PORace-condition creating folder in Python
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORace-condition creating folder in Python
    text
    copied!<p>I have a urllib2 caching module, which sporadically crashes because of the following code:</p> <pre><code>if not os.path.exists(self.cache_location): os.mkdir(self.cache_location) </code></pre> <p>The problem is, by the time the second line is being executed, the folder may exist, and will error:</p> <pre> File ".../cache.py", line 103, in __init__ os.mkdir(self.cache_location) OSError: [Errno 17] File exists: '/tmp/examplecachedir/'</pre> <p>This is because the script is simultaneously launched numerous times, by third-party code I have no control over.</p> <p>The code (before I attempted to fix the bug) can be found <a href="http://github.com/dbr/tvdb_api/commit/e7429cce89fb2406efce6d81336e2ffa01479976" rel="noreferrer">here, on github</a></p> <p>I can't use the <a href="http://docs.python.org/library/tempfile.html#tempfile.mkdtemp" rel="noreferrer">tempfile.mkstemp</a>, as it solves the race condition by using a randomly named directory (<a href="http://svn.python.org/projects/python/trunk/Lib/tempfile.py" rel="noreferrer">tempfile.py source here</a>), which would defeat the purpose of the cache.</p> <p>I don't want to simply discard the error, as the same error Errno 17 error is raised if the folder name exists as a file (a different error), for example:</p> <pre>$ touch blah $ python >>> import os >>> os.mkdir("blah") Traceback (most recent call last): File "", line 1, in OSError: [Errno 17] File exists: 'blah' >>></pre> <p>I cannot using <code>threading.RLock</code> as the code is called from multiple processes.</p> <p>So, I tried writing a simple file-based lock (<a href="http://github.com/dbr/tvdb_api/blob/343df228d519ac5c8895a1e6bdab8d259a64cdb9/cache.py" rel="noreferrer">that version can be found here</a>), but this has a problem: it creates the lockfile one level up, so <code>/tmp/example.lock</code> for <code>/tmp/example/</code>, which breaks if you use <code>/tmp/</code> as a cache dir (as it tries to make <code>/tmp.lock</code>)..</p> <p>In short, I need to cache <code>urllib2</code> responses to disc. To do this, I need to access a known directory (creating it, if required), in a multiprocess safe way. It needs to work on OS X, Linux and Windows.</p> <p>Thoughts? The only alternative solution I can think of is to rewrite the cache module using SQLite3 storage, rather than files.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload