Databases
StackOverflow2013
Postsdbo
POMultiple port scans using external tool, subprocess.Popen and threads
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMultiple port scans using external tool, subprocess.Popen and threads
    primarykey
    data
    text
    <p>I am using a port scanner to scan my subnet. The port scanner unfortunately can scan one port of only one host at a time. Also the scanner has a 1 sec timeout for unreachable hosts. The scanner(being an outside program) has to be run from subprocess.Popen() and to speed it up - so that I may send multiple probes while some previous ones are waiting for replies- I use threads. The problem comes up for a complete /24 subnet scan with large number of threads. SOme of the actually open ports are displayed as closed. I suspect somehow the output gets garbled. Note that this does not occur if I scan fewer hosts or one host at a time</p> <p>The following code is my attempt to create a pool of threads which take an IP address and run 'sequential' port scan for defined port. Once all specified ports are scanned, it picks up next IP from the list. </p> <pre><code> while True: if not thread_queue.empty(): try: hst = ip_iter.next() except StopIteration: break m=thread_queue.get() l=ThreadWork(self,hst,m) l.start() while open_threads != 0: pass </code></pre> <p>Where this fragment sets up thread queue</p> <pre><code> thread_list = [x for x in range(num_threads)] for t in thread_list: thread_queue.put(str(t)) ip_iter=iter(self.final_target) </code></pre> <p>In the ThreadWork function I keep a tab of open threads(since thread_queue.empty proved to be unreliable, I had to use this crude way)</p> <pre><code>class ThreadWork(threading.Thread): def __init__(self,i,hst,thread_no): global open_threads threading.Thread.__init__(self) self.host = hst self.ptr = i self.t = thread_no lock.acquire() open_threads = open_threads + 1 lock.release() def run(self): global thread_queue global open_threads global lock user_log.info("Executing sinfp for IP Address : %s"%self.host) self.ptr.result.append(SinFpRes(self.host,self.ptr.init_ports,self.ptr.all_ports,self.ptr.options,self.ptr.cf)) lock.acquire() open_threads = open_threads - 1 lock.release() thread_queue.put(self.t) </code></pre> <p>The call to SinFpRes creates a result object for one IP and initiate sequential scanning of ports for that IP only. The actual scan per port is as shown</p> <pre><code> com_string = '/usr/local/sinfp/bin/sinfp.pl '+self.options+' -ai '+str(self.ip)+' -p '+str(p) args = shlex.split(com_string) self.result=subprocess.Popen(args,stdout=subprocess.PIPE).communicate()[0] self.parse(p) </code></pre> <p>The parse function then utilises result stored in self.result to store output for that PORT. An aggregate of all the ports is what constitues a scan result for an IP.</p> <p>Calling this code using 10 threads gives accurate o/p (when compared to nmap output). On giving 15 threads an occasional open port is missed. On giving 20 threads, more open ports are missed. On giving 50 threads many ports are missed.</p> <p>P.S. - As a first timer, this code is very convoluted. Apologies to the puritans.</p> <p>P.P.S. - Even a threaded port scan takes 15 minutes for an entire class C subnet with hardly 20 ports scanned. I was wondering if I should move this code to another language and use Python to only parse the results. Could somebody suggest me a language? Note -I am exploring Shell option as shown by S.Lott but manual processing is required before dumping it into file.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USRedBaron
    1. USRedBaron
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POMultiple port scans using external tool, subprocess.Popen and threads
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POMultiple port scans using external tool, subprocess.Popen and threads
      1. USXavier Combelle
      1. VTFavorite
    1. COWhy are you having threads running processes? The OS runs multiple processes like this for you -- without you having to write a bunch of threads.
      singulars
      1. POMultiple port scans using external tool, subprocess.Popen and threads
      1. USS.Lott
    2. CO1. Why don't you use an existing solution such as `nmap`? 2. An asynchronous solutions such as based on http://twistedmatrix.com might be more suitable than threads (to scan 255*64k ports fast) 3. `close_fds=True` wouldn't hurt if you are not on Windows. 4. The problem might not in the `subprocess` but in your scanner, try running it with GNU `parallel` http://savannah.gnu.org/projects/parallel and see if the errors persist. Here's python script https://gist.github.com/717467 from http://stackoverflow.com/questions/4287178/better-multithreaded-use-of-python-subprocess-popen-communicate
      singulars
      1. POMultiple port scans using external tool, subprocess.Popen and threads
      1. USjfs
    3. COActually the project envisages use of more than one scanner. So I have already used nmap as one of the tools. It works fine. For a 'second opinion', I used the scanner 'sinfp'. BTW for 10 threads also the ouput DOES NOT get mixed up. But as I increase the no. of threads, the o/p starts getting murkier. For 50 threads, some alive hosts are also dropped.
      singulars
      1. POMultiple port scans using external tool, subprocess.Popen and threads
      1. USRedBaron
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload