Databases
StackOverflow2013
Postsdbo
POMultiple Python Processes slow
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMultiple Python Processes slow
    primarykey
    data
    text
    <p>I have a python script which goes off and makes a number of HTTP and urllib requests to various domains. </p> <p>We have a huge amount of domains to processes and need to do this as quickly as possible. As HTTP requests are slow (i.e. they could time out of there is no website on the domain) I run a number of the scripts at any one time feeding them from a domains list in the database.</p> <p>The problem I see is over a period of time (a few hours to 24 hours) the scripts all start to slow down and ps -al shows they are sleeping.</p> <p>The servers are very powerful (8 cores, 72GB ram, 6TB Raid 6 etc etc 80MB 2:1 connection) and are never maxed out, i.e. <code>Free -m</code> shows</p> <pre><code>-/+ buffers/cache: 61157 11337 Swap: 4510 195 4315 </code></pre> <p>Top shows between 80-90% idle</p> <p>sar -d shows average 5.3% util</p> <p>and more interestingly iptraf starts off at around 50-60MB/s and ends up 8-10MB/s after about 4 hours.</p> <p>I am currently running around 500 versions of the script on each server (2 servers) and they both show the same problem.</p> <p><code>ps -al</code> shows that most of the python scripts are sleeping which I don't understand why for instance:</p> <pre><code>0 S 0 28668 2987 0 80 0 - 71003 sk_wai pts/2 00:00:03 python 0 S 0 28669 2987 0 80 0 - 71619 inet_s pts/2 00:00:31 python 0 S 0 28670 2987 0 80 0 - 70947 sk_wai pts/2 00:00:07 python 0 S 0 28671 2987 0 80 0 - 71609 poll_s pts/2 00:00:29 python 0 S 0 28672 2987 0 80 0 - 71944 poll_s pts/2 00:00:31 python 0 S 0 28673 2987 0 80 0 - 71606 poll_s pts/2 00:00:26 python 0 S 0 28674 2987 0 80 0 - 71425 poll_s pts/2 00:00:20 python 0 S 0 28675 2987 0 80 0 - 70964 sk_wai pts/2 00:00:01 python 0 S 0 28676 2987 0 80 0 - 71205 inet_s pts/2 00:00:19 python 0 S 0 28677 2987 0 80 0 - 71610 inet_s pts/2 00:00:21 python 0 S 0 28678 2987 0 80 0 - 71491 inet_s pts/2 00:00:22 python </code></pre> <p>There is no sleep state in the script that gets executed so I can't understand why ps -al shows most of them asleep and why they should get slower and slower making less IP requests over time when CPU, memory, disk access and bandwidth are all available in abundance.</p> <p>If anyone could help I would be very grateful.</p> <p>EDIT:</p> <p>The code is massive as I am using exceptions through it to catch diagnostics about the domain, i.e. reasons I can't connect. Will post the code somewhere if needed, but the fundamental calls via HTTPLib and URLLib are straight off the python examples.</p> <p>More info:</p> <p>Both</p> <p>quota -u mysql quota -u root</p> <p>come back with nothing</p> <p>nlimit -n comes back with 1024 have change limit.conf to allow mysql to allow 16000 soft and hard connections and am able to running over 2000 script so far but still still the problem.</p> <h3>SOME PROGRESS</h3> <p>Ok, so I have changed all the limits for the user, ensured all sockets are closed (they were not) and although things are better, I am still getting a slow down although not as bad. </p> <p>Interestingly I have also noticed some memory leak - the scripts use more and more memory the longer they run, however I am not sure what is causing this. I store output data in a string and then print it to the terminal after every iteration, I do clear the string at the end too but could the ever increasing memory be down to the terminal storing all the output?</p> <p>Edit: No seems not - ran up 30 scripts without outputting to terminal and still the same leak. I'm not using anything clever (just strings, HTTPlib and URLLib) - wonder if there are any issues with the python mysql connector...?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USdan360
    1. USdan360
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POMultiple Python Processes slow
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POMultiple Python Processes slow
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POMultiple Python Processes slow
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIt would probably help if you provide some code. How do you do the requests exactly?
      singulars
      1. POMultiple Python Processes slow
      1. USMuhammad Alkarouri
    2. COAre you sure the problem you are facing is not related to your internet connection getting worse upstream?
      singulars
      1. POMultiple Python Processes slow
      1. US6502
    3. COIt shouldn't be - the connection is pretty solid and is 80MB 2:1 both ways - if I kick off say 500 scripts the connection will sit at around 50MB/s for an hour or so and then reduce to 10MB/s over the space of a few hours. If I then kicked off another say 100 it will increase to use 40-50MB's again and then slow over a similar time period. - None of the scripts stop - they just seem to go to sleep as per the ps -al output above.
      singulars
      1. POMultiple Python Processes slow
      1. USdan360
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload