Databases
StackOverflow2013
Postsdbo
POfile.read() multiprocessing and the GIL
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POfile.read() multiprocessing and the GIL
    primarykey
    data
    text
    <p>I've read that certain Python functions implemented in C, which I assume includes file.read(), can release the GIL while they're working and then get it back on completion and by doing so make use of multiple cores if they're available.</p> <p>I'm using multiprocess to parallelize some code and currently I've got three processes, the parent, one child that reads data from a file, and one child that generates a checksum from the data passed to it by the first child process. </p> <p>Now if I'm understanding this right, it seems that creating a new process to read the file as I'm currently doing is uneccessary and I should just call it in the main process. The question is am I understanding this right and will I get better performance with the read kept in the main process or in a separate one?</p> <p>So given my function to read and pipe the data to be processed:</p> <pre><code>def read(file_path, pipe_out): with open(file_path, 'rb') as file_: while True: block = file_.read(block_size) if not block: break pipe_out.send(block) pipe_out.close() </code></pre> <p>I reckon that this will definitely make use of multiple cores, but also introduces some overhead:</p> <pre><code>multiprocess.Process(target=read, args).start() </code></pre> <p>But now I'm wondering if just doing this will also use multiple cores, minus the overhead:</p> <pre><code>read(*args) </code></pre> <p>Any insights anybody has as to which one would be faster and for what reason would be much appreciated!</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USredrah
    1. USredrah
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POfile.read() multiprocessing and the GIL
      1. USGodMan
      1. VTFavorite
    2. VO
      singulars
      1. POfile.read() multiprocessing and the GIL
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POfile.read() multiprocessing and the GIL
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhy don't you just try it out, which is faster and whether multiple cores are used when reading in the main process?
      singulars
      1. POfile.read() multiprocessing and the GIL
      1. USbpgergo
    2. COYou will have to actually create a thread. Just calling ``read(*args)`` won't do any threading (sorry if you know this, it's a bit unclear in your question).
      singulars
      1. POfile.read() multiprocessing and the GIL
      1. USJonas Wielicki
    3. CO@Jonas that was my understanding; no explicit creation of threads means no threads get created... But the research I've done has led me to believe that CPython can in some cases create threads of its own accord to do low level stuff, though they're not exposed to the API. I can't help feeling I've misunderstood something somewhere though which is why I'm interested in getting a better understanding.
      singulars
      1. POfile.read() multiprocessing and the GIL
      1. USredrah
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload