Databases
StackOverflow2013
Postsdbo
POThreading HTTP requests (with proxies)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POThreading HTTP requests (with proxies)
    primarykey
    data
    text
    <p>I've looked at similar questions, but there always seems to be a whole lot of disagreement over the best way to handle threading with HTTP.</p> <p>What I specifically want to do: I'm using Python 2.7, and I want to try and thread HTTP requests (specifically, POSTing something), with a SOCKS5 proxy for each. The code I have already works, but is rather slow since it's waiting for each request (to the proxy server, then the web server) to finish before starting another. Each thread would most likely be making a different request with a different SOCKS proxy.</p> <p>So far I've purely been using urllib2. I looked into modules like PycURL, but it is extremely difficult to install properly with Python 2.7 on Windows, which I want to support and which I am coding on. I'd be willing to use any other module though.</p> <p>I've looked at these questions in particular:</p> <p><a href="https://stackoverflow.com/questions/3472515/python-urllib2-open-is-slow-need-a-better-way-to-read-several-urls">Python urllib2.urlopen() is slow, need a better way to read several urls</a></p> <p><a href="https://stackoverflow.com/questions/5808138/python-example-of-urllib2-asynchronous-threaded-request-using-https">Python - Example of urllib2 asynchronous / threaded request using HTTPS</a></p> <p>Many of the examples received downvotes and arguing. Assuming the commenters are correct, making a client with an asynchronous framework like Twisted sounds like it would be the fastest thing to use. However, I Googled ferociously, and it does not provide any sort of support for SOCKS5 proxies. I'm currently using the Socksipy module, and I could try something like:</p> <pre><code>socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, IP, port) socks.wrapmodule(twisted.web.client) </code></pre> <p>I have no idea if that would work though, and I also don't even know if Twisted is what I really want to use. I could also just go with the threading module and work that into my current urllib2 code, but if that is going to be much slower than Twisted, I may not want to bother. Does anyone have any insight?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USAnorov
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POThreading HTTP requests (with proxies)
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload