Databases
StackOverflow2013
Postsdbo
POPyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
    primarykey
    data
    text
    <p>Basically there seems to be <strong><em>massive</em></strong> confusion/ambiguity over when exactly <a href="http://docs.python.org/3.2/c-api/init.html#PyEval_InitThreads" rel="nofollow noreferrer"><strong><code>PyEval_InitThreads()</code></strong></a> is supposed to be called, and what accompanying API calls are needed. The <a href="http://docs.python.org/3.2/c-api/init.html" rel="nofollow noreferrer">official Python documentation</a> is unfortunately very ambiguous. There are already <a href="https://stackoverflow.com/questions/10625584/embedding-python-in-multithreaded-c-application/10702119">many questions on stackoverflow</a> regarding this topic, and indeed, I've personally already <a href="https://stackoverflow.com/questions/14916815/when-is-pyeval-initthreads-meant-to-be-called">asked a question almost identical</a> to this one, so I won't be particularly surprised if this is closed as a duplicate; but consider that there seems to be no definitive answer to this question. (Sadly, I don't have Guido Van Rossum on speed-dial.)</p> <p>Firstly, let's define the scope of the question here: <strong>what do I want to do?</strong> Well... I want to write a Python extension module in C that will:</p> <ol> <li>Spawn worker threads using the <code>pthread</code> API in C </li> <li>Invoke Python callbacks from within these C threads</li> </ol> <p>Okay, so let's start with the Python docs themselves. The <a href="http://docs.python.org/3.2/c-api/init.html#PyEval_InitThreads" rel="nofollow noreferrer"><strong>Python 3.2 docs</strong></a> say:</p> <blockquote> <p><strong>void PyEval_InitThreads()</strong></p> <p>Initialize and acquire the global interpreter lock. It should be called in the main thread before creating a second thread or engaging in any other thread operations such as PyEval_ReleaseThread(tstate). It is not needed before calling PyEval_SaveThread() or PyEval_RestoreThread().</p> </blockquote> <p>So my understanding here is that:</p> <ol> <li>Any C extension module which spawns threads must call <code>PyEval_InitThreads()</code> from the main thread before any other threads are spawned </li> <li>Calling <code>PyEval_InitThreads</code> locks the GIL</li> </ol> <p>So common sense would tell us that any C extension module which creates threads must call <code>PyEval_InitThreads()</code>, and then release the Global Interpreter Lock. Okay, seems straightforward enough. So <em>prima facie</em>, all that's required would be the following code:</p> <pre><code>PyEval_InitThreads(); /* initialize threading and acquire GIL */ PyEval_ReleaseLock(); /* Release GIL */ </code></pre> <p>Seems easy enough... but unfortunately, the Python 3.2 docs <strong>also</strong> say that <a href="http://docs.python.org/3.2/c-api/init.html#PyEval_ReleaseLock" rel="nofollow noreferrer"><strong><code>PyEval_ReleaseLock</code></strong> has been <strong>deprecated</strong></a>. Instead, we're supposed to use <a href="http://docs.python.org/3.2/c-api/init.html#PyEval_SaveThread" rel="nofollow noreferrer"><strong><code>PyEval_SaveThread</code></strong></a> in order to release the GIL:</p> <blockquote> <p><strong>PyThreadState* PyEval_SaveThread()</strong></p> <p>Release the global interpreter lock (if it has been created and thread support is enabled) and reset the thread state to NULL, returning the previous thread state (which is not NULL). If the lock has been created, the current thread must have acquired it.</p> </blockquote> <p>Er... okay, so I guess a C extension module needs to say:</p> <pre><code>PyEval_InitThreads(); PyThreadState* st = PyEval_SaveThread(); </code></pre> <p><br> Indeed, this is exactly what <a href="https://stackoverflow.com/questions/10625584/embedding-python-in-multithreaded-c-application/10702119#10702119"><strong>this stackoverflow answer</strong></a> says. Except when I actually <em>try</em> this in practice, the Python interpreter immediately seg-faults when I import the extension module. &nbsp;&nbsp;&nbsp;Nice. <br> <br></p> <hr> <p>Okay, so now I'm giving up on the official Python documentation and turning to Google. So, <strong><a href="http://dandesousa.com/2011/12/11/boost-python-calling-back-into-python-from-native-c-threads/" rel="nofollow noreferrer">this random blog</a></strong> claims all you need to do from an extension module is to call <code>PyEval_InitThreads()</code>. Of course, the documentation claims that <code>PyEval_InitThreads()</code> acquires the GIL, and indeed, a <a href="https://github.com/certik/python-3.2/blob/master/Python/ceval.c#L304" rel="nofollow noreferrer"><strong><em>quick inspection of the source code for <code>PyEval_InitThreads()</code> in <code>ceval.c</code></em></strong></a> reveals that it does indeed call the internal function <code>take_gil(PyThreadState_GET());</code> </p> <p>So <code>PyEval_InitThreads()</code> <em>definitely</em> acquires the GIL. I would think then that you would absolutely need to somehow release the GIL after calling <code>PyEval_InitThreads()</code>. &nbsp;&nbsp;<strong><em>But how?</em></strong> <code>PyEval_ReleaseLock()</code> is deprecated, and <code>PyEval_SaveThread()</code> just inexplicably seg-faults.</p> <p>Okay... so maybe for some reason which is currently beyond my understanding, a C extension module <em>doesn't</em> need to release the GIL. I tried that... and, as expected, as soon as another thread attempts to acquire the GIL (using <a href="http://docs.python.org/3.2/c-api/init.html#PyGILState_Ensure" rel="nofollow noreferrer"><strong>PyGILState_Ensure</strong></a>), the program hangs from a deadlock. So yeah... you <em>really do</em> need to release the GIL after calling <code>PyEval_InitThreads()</code>.</p> <p>So again, the question is: <strong><em>how do you release the GIL after calling <a href="http://docs.python.org/3.2/c-api/init.html#PyEval_InitThreads" rel="nofollow noreferrer"><code>PyEval_InitThreads()</code></a>?</em></strong></p> <p>And more generally: <strong>what exactly does a C-extension module have to do to be able to safely invoke Python code from worker C-threads?</strong></p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCœur
    1. USChannel72
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTDuplicate
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
      1. This table or related slice is empty.
      1. VTUpMod
    1. COrelated: [Python code calls C library that create OS threads, which eventually call Python callbacks](http://stackoverflow.com/questions/12116390/python-code-calls-c-library-that-create-os-threads-which-eventually-call-python). See the example [`c_extension` module there](https://gist.github.com/zed/3471699) (its purpose is to trigger an error in `threading` while being "correct" to expose a bug in `threading` implementation. It fails to trigger an error on Python 3).
      singulars
      1. POPyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
      1. USjfs
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload