Databases
StackOverflow2013
Postsdbo
POIntermittent issues while accessing external http services using gevent
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIntermittent issues while accessing external http services using gevent
    primarykey
    data
    text
    <p>First off, the versions:</p> <ul> <li>gevent - v0.13.7 </li> <li>gunicorn - v0.14.2 </li> <li>requests - 0.11.2</li> </ul> <p>We recently upgraded our servers that are running behind gunicorn to use the gevent asynchronous workers instead of just normal sync workers. Everything works great, but we're now experiencing an issue when attempting to access a 3rd party service over http and I just have no idea how to track down what might be the issue.</p> <p>A brief stack trace looks like the following:</p> <pre><code>File "/home/deploy/.virtualenvs/bapp/lib/python2.7/site-packages/requests/sessions.py", line 295, in post return self.request('post', url, data=data, **kwargs) File "/home/deploy/.virtualenvs/bapp/lib/python2.7/site-packages/requests/sessions.py", line 252, in request r.send(prefetch=prefetch) File "/home/deploy/.virtualenvs/bapp/lib/python2.7/site-packages/requests/models.py", line 625, in send raise ConnectionError(sockerr) ConnectionError: [Errno 66] unknown </code></pre> <p>Another different stack trace but we think it's the same issue:</p> <pre><code>File "/home/deploy/.virtualenvs/bapp/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 94, in connect sock = socket.create_connection((self.host, self.port), self.timeout) File "/home/deploy/.virtualenvs/bapp/lib/python2.7/site-packages/gevent/socket.py", line 637, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): File "/home/deploy/.virtualenvs/bapp/lib/python2.7/site-packages/gevent/socket.py", line 769, in getaddrinfo raise DNSError: [Errno 66] unknown </code></pre> <p>At first, I thought it could be potentially something related to a <code>libevent-dns</code>, from this <a href="https://groups.google.com/forum/?fromgroups=#!topic/gevent/eC0hj3LJ7_E" rel="nofollow noreferrer">google groups issue</a>. I checked our <code>/etc/resolv.conf</code>, and there is only one dns resolution service:</p> <pre><code>[me@host:~]$ cat /etc/resolv.conf ; generated by /sbin/dhclient-script nameserver 10.3.0.2 </code></pre> <p>I looked up what ERRNO66 is: <a href="https://github.com/libevent/libevent/blob/master/include/event2/dns.h#L162" rel="nofollow noreferrer">https://github.com/libevent/libevent/blob/master/include/event2/dns.h#L162</a>,"/** An unknown error occurred */". I'm not having much luck finding that helpful..sounds like it couldn't talk to the dns server?</p> <p>I thought it might have to do something with <code>python-requests</code>, see <a href="https://stackoverflow.com/questions/9501663/how-enable-requests-async-mode">how enable requests async mode?</a> since <code>python-requests</code> depends on <code>urllib3</code>, which is implemented in terms of <code>httplib</code>; but, it turns out the author of <code>gevent</code> removed the <code>httplib</code> patch in <a href="https://bitbucket.org/denis/gevent/changeset/d50a522a264bf28b6facb809731d3b41c5a49f51#chg-gevent/monkey.py" rel="nofollow noreferrer">this</a> commit earlier this year without any comments as to why.</p> <p>Does anyone have any ideas on how to approach debugging this issue or might shed some light on what's happening here? </p> <p>Thanks in advance!</p> <h1>Update - 12:50PM PDT</h1> <p>After some conversations on freenode, the #gevent and the #gunicorn channel seem to shed some more insight:</p> <h3>#gevent</h3> <ul> <li>gevent v0.13.7 still supports the <code>patch_all</code> with <code>httplib=True</code></li> <li>I asked if "it make sense to patch it?", the response was no.</li> <li>Recommendation to use gevent 1.0 (even if it's beta).</li> <li><p>quote from @schmir:</p> <blockquote> <p>"patch httplib uses libevent http client library. I don't trust libevent. my advice would have been to turn it off, if you used it"</p> </blockquote></li> </ul> <h3>#gunicorn</h3> <ul> <li><code>&lt;Damianz&gt;</code> What's your platform? I've seen that issue appear on windows boxes where it tries ipv6 and just fails life.. (I'm on CentOS 5)</li> <li><code>&lt;dmishe&gt;</code> I've seen similar on mac, looks like gevent beta fixed it</li> </ul> <p>Sounds like the general advice is to ditch gevent v0.13.7 and upgrade to gevent 1.0b. </p> <p>I'll follow up on if that fixes this issue. Meanwhile, anyone that can shed advice, I'd much appreciate it.</p> <h1>Update #2 - 4 days in production, 1:15PM PDT</h1> <p>Looks like the upgrade to <code>gevent</code> has solved this issue -- I'll add my answer and accept it if no one else chimes in, but only after a week without incidents in production.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USMahmoud Abdelkader
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POIntermittent issues while accessing external http services using gevent
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POIntermittent issues while accessing external http services using gevent
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POIntermittent issues while accessing external http services using gevent
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThe last time i tested gevent 1.0 (libev) i run into issue that example of resolving 300 hostnames run about 2 seconds while 0.13 (libevent) version run 0.1sec or so. 0.13 using libevent resolving mechanism and caching while 1.0 uses c-ares. As for your question: if that is reproducible - you could try stracing.
      singulars
      1. POIntermittent issues while accessing external http services using gevent
      1. USTurnaev Evgeny
    2. COThe upgrade to gevent 1.0b(libev) seems to have eliminated the issues with DNSError. The issue is not reproducible, but actually quite intermittent and very disruptive to our business. I'll keep an eye out for these performance issues, meanwhile, I suggest you notify gevent's team so that they're aware of your issue going forward.
      singulars
      1. POIntermittent issues while accessing external http services using gevent
      1. USMahmoud Abdelkader
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload