Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <h3><em>edit</em>: breaking news; see below · · · ······</h3> <p>I decided recently that I wanted to do something pretty similar, and what came out of it was the <a href="http://pypi.python.org/pypi/pqueue_fetcher" rel="nofollow noreferrer">pqueue_fetcher</a> module. It ended up being mainly a learning endeavour: I learned, among other things, that it's almost certainly better to use something like <a href="http://twistedmatrix.com" rel="nofollow noreferrer">twisted</a> than to try to kill Python threads with any sort of reliability.</p> <p>That being said, there's code in that module that more or less answers your question. It basically consists of a class whose objects can be set up to get locations from a priority queue and feed them into a <code>fetch</code> function that's supplied at object instantiation. If the location's resources get successfully received before their thread is killed, they get forwarded on to the <code>results</code> queue; otherwise they're returned to the <code>locations</code> queue with a downgraded priority. Success is determined by a passed-in function that defaults to <code>bool</code>.</p> <p>Along the way I ended up creating the <a href="http://pypi.python.org/pypi/terminable_thread" rel="nofollow noreferrer">terminable_thread</a> module, which just packages the most mature variation I could find of the code you linked to as <code>InterruptableThread</code>. It also adds a fix for 64-bit machines, which I needed in order to use that code on my ubuntu box. <code>terminable_thread</code> is a dependency of <code>pqueue_fetcher</code>.</p> <p>Probably the biggest stumbling block I hit is that raising an asynchronous exception as do <code>terminable_thread</code>, and the <code>InterruptableThread</code> you mentioned, can have some weird results. In the test suite for <code>pqueue_fetcher</code>, the <code>fetch</code> function blocks by calling <code>time.sleep</code>. I found that if a thread is <code>terminate()</code>d while so blocking, and the <code>sleep</code> call is the last (or not even the last) statement in a nested try block, execution will actually bounce to the <code>except</code> clause of the <strong><em>outer</em></strong> try block, <em>even if the inner one has an <code>except</code> matching the raised exception</em>. I'm still sort of shaking my head in disbelief, but there's a test case in <code>pqueue_fetcher</code> that reenacts this. I believe "leaky abstraction" is the correct term here.</p> <p>I wrote a hacky workaround that just does some random thing (in this case getting a value from a generator) to break up the "atomicity" (not sure if that's actually what it is) of that part of the code. This workaround can be overridden via the <code>fission</code> parameter to <code>pqueue_fetcher.Fetcher</code>. It (i.e. the default one) seems to work, but certainly not in any way that I would consider particularly reliable or portable.</p> <p>So my call after discovering this interesting piece of data was to heretofore avoid using this technique (i.e. calling <code>ctypes.pythonapi.PyThreadState_SetAsyncExc</code>) altogether.</p> <p>In any case, this still won't work if you need to guarantee that any request whose entire data set has been received (and i.e. acknowledged to the server) gets forwarded on to <code>results</code>. In order to be sure of that, you have to guarantee that the bit that does that last network transaction and the forwarding is guarded from being interrupted, without guarding the entire retrieval operation from being interrupted (since this would prevent timeouts from working..). And in order to do that you need to basically rewrite the retrieval operation (i.e. the socket code) to be aware of whichever exception you're going to raise with <code>terminable_thread.Thread.raise_exc</code>.</p> <p>I've yet to learn twisted, but being the Premier Python Asynchronous Networking Framework©™®, I expect it must have some elegant or at least workable way of dealing with such details. I'm hoping it provides a parallel way to implement fetching from non-network sources (e.g. a local filestore, or a DB, or an etc.), since I'd like to build an app that can glean data from a variety of sources in a medium-agnostic way. </p> <p>Anyhow, if you're still intent on trying to work out a way to manage the threads yourself, you can perhaps learn from my efforts. Hope this helps.</p> <h3>· · · · ······ this just in:</h3> <p>I've realized that the tests that I thought had stabilized have actually not, and are giving inconsistent results. This appears to be related to the issues mentioned above with exception handling and the use of the <code>fission</code> function. I'm not really sure what's going on with it, and don't plan to investigate in the immediate future unless I end up having a need to actually do things this way.</p>
    singulars
    1. This table or related slice is empty.
    1. POPython threading passing statuses
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USintuited
    1. USintuited
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POPython threading passing statuses
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload