Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>As far as I know, <code>pp</code> doesn't have any such feature in its API. </p> <p>If you used the stdlib modules instead, that would make your life a lot easier—e.g., <a href="http://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool" rel="nofollow"><code>multiprocessing.Pool</code></a> takes an <code>initializer</code> argument, which you could use to initialize a database for each process, which would then be available as a variable that each task could use.</p> <p>However, there is a relatively easy workaround.</p> <p>Each process has a unique (at least while it's running) process ID.* In Python, you can access the process ID of the current process with <a href="http://docs.python.org/3/library/os.html#os.getpid" rel="nofollow"><code>os.getpid()</code></a>. So, in each task, you can do something like this:</p> <pre><code>dbname = 'database{}'.format(os.getpid()) </code></pre> <p>Then use <code>dbname</code> to open/create the database. I don't know whether by "database" you mean a <code>dbm</code> file, a <code>sqlite3</code> file, a database on a MySQL server, or what. You may need to, e.g., create a <code>tempfile.TemporaryDirectory</code> in the parent, pass it to all of the children, and have them <code>os.path.join</code> it to the dbname (so after all the children are done, you can grab everything in <code>os.listdir(the_temp_dir)</code>).</p> <hr> <p>The problem with this is that if <code>pp.Server</code> restarts one of the processes, you'll end up with 4 databases instead of 3. Probably not a huge deal, but your code should deal with that possibility. (IIRC, <code>pp.Server</code> usually doesn't restart the processes unless you pass <code>restart=True</code>, but it <em>may</em> do so if, e.g., one of them crashes.)</p> <p>But what if (as seems to be the case) you're actually running each task in a brand-new process, rather than using a pool of 3 processes? Well, then you're going to end up with as many databases as there are processes, which probably isn't what you want. Your real problem here is that you're not using a pool of 3 processes, which is what you ought to fix. But are there other ways you could get what you want? Maybe.</p> <p>For example, let's say you created three locks, one for each database, maybe as lockfiles. Then, each task could do this pseudocode:</p> <pre><code>for i, lockfile in enumerate(lockfiles): try: with lockfile: do stuff with databases[i] break except AlreadyLockedError: pass else: assert False, "oops, couldn't get any of the locks" </code></pre> <p>If you can actually lock the databases themselves (with an flock, or with some API for the relevant database, etc.) things are even easier: just try to connect to them in turn until one of them succeeds.</p> <p>As long as your code isn't actually segfaulting or the like,** if you're actually never running more than 3 tasks at a time, there's no way all 3 lockfiles could be locked, so you're guaranteed to get one.</p> <hr> <p>* This isn't quite true, but it's true enough for your purposes. For example, on Windows, each process has a unique <code>HANDLE</code>, and if you <em>ask</em> for its <code>pid</code> one will be generated if it didn't already have one. And on some *nixes, each <em>thread</em> has a unique <em>thread</em> ID, and the process's <code>pid</code> is the thread ID of the first thread. And so on. But as far as your code can tell, each of your processes has a unique <code>pid</code>, which is what matters. </p> <p>** Even if your code <em>is</em> crashing, you can deal with that, it's just more complicated. For example, use pidfiles instead of empty lockfiles. Get a read lock on the pidfile, then try to upgrade to a write lock. If it fails, read the pid from the file, and check whether any such process exists (e.g., on *nix, if <code>os.kill(pid, 0)</code> raises, there is no such process), and if so forcibly break the lock. Either way, now you've got a write lock, so write your pid to the file.</p>
    singulars
    1. This table or related slice is empty.
    1. POidentifying a processor core or worker id parallel python
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USabarnert
    1. USabarnert
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POidentifying a processor core or worker id parallel python
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COI tried process ID, but each job gets a different pid, whereas I only want as many db's as there are works (not jobs). I then tried to look back at a specific core from the pid but couldn't find a way to do this either. The db is a geographic one, I can't reliable check if it's locked which is why I want to run separate databses.
      singulars
      1. PO
      1. USSteven
    2. CO@Steven: If each job gets a different PID, then each job is running in a different process, rather than them all running on a pool of three long-lived processes. So there is absolutely no way to do what you want, unless you change that. I'll update the answer with some alternatives that might work, but really, if your question assumes pooling, you have to get your tasks pooled.
      singulars
      1. PO
      1. USabarnert
    3. COThanks for your thoughts, definitely helped my thinking. I think I could get where I want in pp by creating my own lock files in a common spot, then staggering the initial job distribution to ensure the locks are created.
      singulars
      1. PO
      1. USSteven
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload