Databases
StackOverflow2013
Postsdbo
POIs this Python producer-consumer lockless approach thread-safe?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs this Python producer-consumer lockless approach thread-safe?
    text
    copied!<p>I recently wrote a program that used a simple producer/consumer pattern. It initially had a bug related to improper use of threading.Lock that I eventually fixed. But it made me think whether it's possible to implement producer/consumer pattern in a lockless manner.</p> <p>Requirements in my case were simple:</p> <ul> <li>One producer thread.</li> <li>One consumer thread.</li> <li>Queue has place for only one item.</li> <li>Producer can produce next item before the current one is consumed. The current item is therefore lost, but that's OK for me.</li> <li>Consumer can consume current item before the next one is produced. The current item is therefore consumed twice (or more), but that's OK for me.</li> </ul> <p>So I wrote this:</p> <pre><code>QUEUE_ITEM = None # this is executed in one threading.Thread object def producer(): global QUEUE_ITEM while True: i = produce_item() QUEUE_ITEM = i # this is executed in another threading.Thread object def consumer(): global QUEUE_ITEM while True: i = QUEUE_ITEM consume_item(i) </code></pre> <p>My question is: Is this code thread-safe?</p> <p>Immediate comment: this code isn't really lockless - I use CPython and it has GIL.</p> <p>I tested the code a little and it seems to work. It translates to some LOAD and STORE ops which are atomic because of GIL. But I also know that <code>del x</code> operation isn't atomic when x implements <code>__del__</code> method. So if my item has a <code>__del__</code> method and some nasty scheduling happens, things may break. Or not?</p> <p>Another question is: What kind of restrictions (for example on produced items' type) do I have to impose to make the above code work fine?</p> <p>My questions are only about theoretical possibility to exploit CPython's and GIL's quirks in order to come up with lockless (i.e. no locks like threading.Lock explicitly in code) solution.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload