StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Okay this is an answer to my own post. Well done me =D</p> <p>Produces about a 150% increase in speed on my system going from a single core thread to quad core multiprocessing.</p> <pre><code>import multiprocessing, time, psycopg2 class Consumer(multiprocessing.Process): def __init__(self, task_queue, result_queue): multiprocessing.Process.__init__(self) self.task_queue = task_queue self.result_queue = result_queue def run(self): proc_name = self.name while True: next_task = self.task_queue.get() if next_task is None: print 'Tasks Complete' self.task_queue.task_done() break answer = next_task() self.task_queue.task_done() self.result_queue.put(answer) return class Task(object): def __init__(self, a): self.a = a def __call__(self): pyConn = psycopg2.connect("dbname='geobase_1' host = 'localhost'") pyConn.set_isolation_level(0) pyCursor1 = pyConn.cursor() procQuery = 'UPDATE city SET gid_fkey = gid FROM country WHERE ST_within((SELECT the_geom FROM city WHERE city_id = %s), country.the_geom) AND city_id = %s' % (self.a, self.a) pyCursor1.execute(procQuery) print 'What is self?' print self.a return self.a def __str__(self): return 'ARC' def run(self): print 'IN' if __name__ == '__main__': tasks = multiprocessing.JoinableQueue() results = multiprocessing.Queue() num_consumers = multiprocessing.cpu_count() * 2 consumers = [Consumer(tasks, results) for i in xrange(num_consumers)] for w in consumers: w.start() pyConnX = psycopg2.connect("dbname='geobase_1' host = 'localhost'") pyConnX.set_isolation_level(0) pyCursorX = pyConnX.cursor() pyCursorX.execute('SELECT count(*) FROM cities WHERE gid_fkey IS NULL') temp = pyCursorX.fetchall() num_job = temp[0] num_jobs = num_job[0] pyCursorX.execute('SELECT city_id FROM city WHERE gid_fkey IS NULL') cityIdListTuple = pyCursorX.fetchall() cityIdList = [] for x in cityIdListTuple: cityIdList.append(x[0]) for i in xrange(num_jobs): tasks.put(Task(cityIdList[i - 1])) for i in xrange(num_consumers): tasks.put(None) while num_jobs: result = results.get() print result num_jobs -= 1 </code></pre> <p>Now I have another question which I have posted here:</p> <p><a href="https://stackoverflow.com/questions/7555680/create-db-connection-and-maintain-on-multiple-processes-multiprocessing">Create DB connection and maintain on multiple processes (multiprocessing)</a></p> <p>Hopefully we can get rid of some overhead and speed this baby up even more.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload