StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Welcome the world of concurrent programming.</p> <p>What Python can (and can't) do depends on two things.</p> <ol> <li><p>What the OS can (and can't) do. Most OS's allocate processes to cores. To use 4 cores, you need to break your problem into four processes. This is easier than it sounds. Sometimes.</p></li> <li><p>What the underlying C libraries can (and can't) do. If the C libraries expose features of the OS AND the OS exposes features of the hardware, you're solid. </p></li> </ol> <p>To break a problem into multiple processes -- especially in GNU/Linux -- is easy. Break it into a multi-step pipeline.</p> <p>In the case of summing a million numbers, think of the following shell script. Assuming some hypothetical <code>sum.py</code> program that sums either a range of numbers or a list of numbers on stdin.</p> <p>( sum.py 0 500000 & sum.py 50000 1000000 ) | sum.py</p> <p>This would have 3 concurrent processes. Two are doing sums of a lot of numbers, the third is summing two numbers. </p> <p>Since the GNU/Linux shells and the OS already handle some parts of concurrency for you, you can design simple (very, very simple) programs that read from stdin, write to stdout, and are designed to do small parts of a large job.</p> <p>You can try to reduce the overheads by using <a href="http://docs.python.org/library/subprocess.html" rel="noreferrer">subprocess</a> to build the pipeline instead of allocating the job to the shell. You may find, however, that the shell builds pipelines very, very quickly. (It was written directly in C and makes direct OS API calls for you.)</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload