Databases
StackOverflow2013
Postsdbo
POWhich is best in Python: urllib2, PycURL or mechanize?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhich is best in Python: urllib2, PycURL or mechanize?
    text
    copied!<p>Ok so I need to download some web pages using Python and did a quick investigation of my options.</p> <p>Included with Python:</p> <p><a href="http://docs.python.org/library/urllib.html" rel="noreferrer">urllib</a> - seems to me that I should use urllib2 instead. urllib has no cookie support, HTTP/FTP/local files only (no SSL)</p> <p><a href="http://docs.python.org/library/urllib2.html" rel="noreferrer">urllib2</a> - complete HTTP/FTP client, supports most needed things like cookies, does not support all HTTP verbs (only GET and POST, no TRACE, etc.)</p> <p>Full featured:</p> <p><a href="http://wwwsearch.sourceforge.net/mechanize/" rel="noreferrer">mechanize</a> - can use/save Firefox/IE cookies, take actions like follow second link, actively maintained (0.2.5 released in March 2011)</p> <p><a href="http://pycurl.sourceforge.net/" rel="noreferrer">PycURL</a> - supports everything curl does (FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET, DICT, FILE and LDAP), bad news: not updated since Sep 9, 2008 (7.19.0)</p> <p>New possibilities:</p> <p><a href="http://code.google.com/p/urllib3/" rel="noreferrer">urllib3</a> - supports connection re-using/pooling and file posting</p> <p>Deprecated (a.k.a. use urllib/urllib2 instead):</p> <p><a href="http://docs.python.org/library/httplib.html" rel="noreferrer">httplib</a> - HTTP/HTTPS only (no FTP)</p> <p><a href="http://code.google.com/p/httplib2/" rel="noreferrer">httplib2</a> - HTTP/HTTPS only (no FTP)</p> <p>The first thing that strikes me is that urllib/urllib2/PycURL/mechanize are all pretty mature solutions that work well. mechanize and PycURL ship with a number of Linux distributions (e.g. Fedora 13) and BSDs so installation is a non issue typically (so that's good). </p> <p>urllib2 looks good but I'm wondering why PycURL and mechanize both seem very popular, is there something I am missing (i.e. if I use urllib2 will I paint myself in to a corner at some point?). I'd really like some feedback on the pros/cons of these things so I can make the best choice for myself. </p> <p>Edit: added note on verb support in urllib2</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload