StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>You can't make Python faster. But you can make your Python application faster.</p> <p>Principle 1: Do Less.</p> <p>You can't do less input parsing <em>over all</em> but you can do less input parsing in the process that's also reading the socket and doing everything else with the data.</p> <p>Generally, do this.</p> <p>Break your application into a pipeline of discrete steps.</p> <ol> <li><p>Read the socket, break into fields, create a named tuple, write the tuple to a pipe with something like <code>pickle</code>.</p></li> <li><p>Read a pipe (with <code>pickle</code>) to construct the named tuple, do some processing, write to another pipe.</p></li> <li><p>Read a pipe, do some processing, write to a file or something.</p></li> </ol> <p>Each of these three processes, connected with OS pipes, runs concurrently. That means that the first process is reading the socket and make tuples while the second process is consuming tuples and doing calculations while the third process is doing calculations and writing a file.</p> <p>This kind of pipeline maximizes what your CPU can do. Without too many painful tricks.</p> <p>Reading and writing to pipes is trivial, since linux assures you that sys.stdin and sys.stdout will be pipes when the shell creates the pipeline.</p> <p>Before doing anything else, break your program into pipeline stages.</p> <p>proc1.py</p> <pre><code>import cPickle from collections import namedtuple ChanVal= namedtuple( 'ChanVal', ['eventTime','eventTimeExact', 'other_clock', ... ] ) for line socket: c= ChanVal( **line.split(',') ) cPickle.dump( sys.stdout ) </code></pre> <p>proc2.py</p> <pre><code>import cPickle from collections import namedtuple ChanVal= namedtuple( 'ChanVal', ['eventTime','eventTimeExact', 'other_clock', ... ] ) while True: item = cPickle.load( sys.stdin ) # processing cPickle.dump( sys.stdout ) </code></pre> <p>This idea of processing namedtuples through a pipeline is very scalable.</p> <pre><code>python proc1.py | python proc2.py </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload