StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POWhat are the advantages of multithreaded programming in Python?
text
Body
copied!<p>When I hear about multithreaded programming, I think about the opportunity to accelerate my program, but it is not?</p> <pre><code>import eventlet from eventlet.green import socket from iptools import IpRangeList class Scanner(object): def __init__(self, ip_range, port_range, workers_num): self.workers_num = workers_num or 1000 self.ip_range = self._get_ip_range(ip_range) self.port_range = self._get_port_range(port_range) self.scaned_range = self._get_scaned_range() def _get_ip_range(self, ip_range): return [ip for ip in IpRangeList(ip_range)] def _get_port_range(self, port_range): return [r for r in range(*port_range)] def _get_scaned_range(self): for ip in self.ip_range: for port in self.port_range: yield (ip, port) def scan(self, address): try: return bool(socket.create_connection(address)) except: return False def run(self): pool = eventlet.GreenPool(self.workers_num) for status in pool.imap(self.scan, self.scaned_range): if status: yield True def run_std(self): for status in map(self.scan, self.scaned_range): if status: yield True if __name__ == '__main__': s = Scanner(('127.0.0.1'), (1, 65000), 100000) import time now = time.time() open_ports = [i for i in s.run()] print 'Eventlet time: %s (sec) open: %s' % (now - time.time(), len(open_ports)) del s s = Scanner(('127.0.0.1'), (1, 65000), 100000) now = time.time() open_ports = [i for i in s.run()] print 'CPython time: %s (sec) open: %s' % (now - time.time(), len(open_ports)) </code></pre> <p>and <strong>results:</strong></p> <pre><code>Eventlet time: -4.40343403816 (sec) open: 2 CPython time: -4.48356699944 (sec) open: 2 </code></pre> <p>And my question is, if I run this code is not on my laptop but on the server and set more value of workers it will run faster than the CPython's version? What are the advantages of threads?</p> <p><strong>ADD:</strong> And so I rewrite app with use original cpython's threads</p> <pre><code>import socket from threading import Thread from Queue import Queue from iptools import IpRangeList class Scanner(object): def __init__(self, ip_range, port_range, workers_num): self.workers_num = workers_num or 1000 self.ip_range = self._get_ip_range(ip_range) self.port_range = self._get_port_range(port_range) self.scaned_range = [i for i in self._get_scaned_range()] def _get_ip_range(self, ip_range): return [ip for ip in IpRangeList(ip_range)] def _get_port_range(self, port_range): return [r for r in range(*port_range)] def _get_scaned_range(self): for ip in self.ip_range: for port in self.port_range: yield (ip, port) def scan(self, q): while True: try: r = bool(socket.create_conection(q.get())) except Exception: r = False q.task_done() def run(self): queue = Queue() for address in self.scaned_range: queue.put(address) for i in range(self.workers_num): worker = Thread(target=self.scan,args=(queue,)) worker.setDaemon(True) worker.start() queue.join() if __name__ == '__main__': s = Scanner(('127.0.0.1'), (1, 65000), 5) import time now = time.time() s.run() print time.time() - now </code></pre> <p>and result is</p> <pre><code> Cpython's thread: 1.4 sec </code></pre> <p>And I think this is a very good result. I take as a standard nmap scanning time:</p> <pre><code>$ nmap 127.0.0.1 -p1-65000 Starting Nmap 5.21 ( http://nmap.org ) at 2012-10-22 18:43 MSK Nmap scan report for localhost (127.0.0.1) Host is up (0.00021s latency). Not shown: 64986 closed ports PORT STATE SERVICE 53/tcp open domain 80/tcp open http 443/tcp open https 631/tcp open ipp 3306/tcp open mysql 6379/tcp open unknown 8000/tcp open http-alt 8020/tcp open unknown 8888/tcp open sun-answerbook 9980/tcp open unknown 27017/tcp open unknown 27634/tcp open unknown 28017/tcp open unknown 39900/tcp open unknown Nmap done: 1 IP address (1 host up) scanned in 0.85 seconds </code></pre> <p>And my question is now: how threads implemented in Eventlet as I can understand this is not threads but something special for Eventlet and why they dont speed up tasks?</p> <p>Eventlet is used by many of the major projects like OpenStack and etc. But why? Just do the heavy queries to a DB in asynchronous manner or something else?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload