StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>I had done some similar task in the past with Machine Learning and Data Mining. Using <code>multiprocessing</code> in your case may not be that difficult of a task. It depends on how tolerant you are keen on making the program, you can use a <em>Threaded Pool</em> pattern. My personal favourite is <em>Producer - Consumer</em> pattern using <code>Queue</code>, this design can handle a variety of complex task. Here is a sample toy program using <code>multiprocessing</code>:</p> <pre><code>import multiprocessing from multiprocessing import Queue, Process from Queue import Empty as QueueEmpty # Assuming this text is very very very very large text="Here I am writing some nonsense\nBut people will read\n..." def read(q): """Read the text and put in a queue""" for line in text.split("\n"): q.put(line) def work(qi, qo): """Put the line into the queue out""" while True: try: data = qi.get(timeout = 1) # Timeout after 1 second qo.put(data) except QueueEmpty: return # Exit when all work is done except: raise # Raise all other errors def join(q): """Join all the output queue and write to a text file""" f = open("file.txt", w) while True: try: f.write(q.get(timeout=1)) except QueueEmpty: f.close() return except: raise def main(): # Input queue qi = Queue() # Output queue qo = Queue() # Start the producer Process(target = read, args = (qi, )).start() # Start 8 consumers for i in range(8): Process(target = work, args = (qi, qo, )).start() # Final process to handle the queue out Process(target = join, args = (qo, )).start() </code></pre> <p>Type this from memory so if there is any error, please correct. :)</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload