StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO302s and losing cookies with urllib2
text
Body
copied!<p>I am using liburl2 with CookieJar / HTTPCookieProcessor in an attempt to simulate a login to a page to automate an upload.</p> <p>I've seen some questions and answers on this, but nothing which solves my problem. I am losing my cookie when I simulate the login which ends up at a 302 redirect. The 302 response is where the cookie gets set by the server, but urllib2 HTTPCookieProcessor does not seem to save the cookie during a redirect. I tried creating a HTTPRedirectHandler class to ignore the redirect, but that didn't seem to do the trick. I tried referencing the CookieJar globally to handle the cookies from the HTTPRedirectHandler, but 1. This didn't work (because I was handling the header from the redirector, and the CookieJar function that I was using, extract_cookies, needed a full request) and 2. It's an ugly way to handle it.</p> <p>I probably need some guidance on this as I'm fairly green with Python. I think I'm mostly barking up the right tree here, but maybe focusing on the wrong branch.</p> <pre><code>cj = cookielib.CookieJar() cookieprocessor = urllib2.HTTPCookieProcessor(cj) class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler): def http_error_302(self, req, fp, code, msg, headers): global cj cookie = headers.get("set-cookie") if cookie: # Doesn't work, but you get the idea cj.extract_cookies(headers, req) return urllib2.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers) http_error_301 = http_error_303 = http_error_307 = http_error_302 cookieprocessor = urllib2.HTTPCookieProcessor(cj) # Oh yeah. I'm using a proxy too, to follow traffic. proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8888'}) opener = urllib2.build_opener(MyHTTPRedirectHandler, cookieprocessor, proxy) </code></pre> <p>Addition: I had tried using mechanize as well, without success. This is probably a new question, but I'll pose it here since it is the same ultimate goal:</p> <p>This simple code using mechanize, when used with a 302 emitting url (http://fxfeeds.mozilla.com/firefox/headlines.xml) -- note that the same behavior occurs when not using set_handle_robots(False). I just wanted to ensure that wasn't it:</p> <pre><code>import urllib2, mechanize browser = mechanize.Browser() browser.set_handle_robots(False) opener = mechanize.build_opener(*(browser.handlers)) r = opener.open("http://fxfeeds.mozilla.com/firefox/headlines.xml") </code></pre> <p>Output:</p> <pre><code>Traceback (most recent call last): File "redirecttester.py", line 6, in <module> r = opener.open("http://fxfeeds.mozilla.com/firefox/headlines.xml") File "build/bdist.macosx-10.6-universal/egg/mechanize/_opener.py", line 204, in open File "build/bdist.macosx-10.6-universal/egg/mechanize/_urllib2_fork.py", line 457, in http_response File "build/bdist.macosx-10.6-universal/egg/mechanize/_opener.py", line 221, in error File "build/bdist.macosx-10.6-universal/egg/mechanize/_urllib2_fork.py", line 332, in _call_chain File "build/bdist.macosx-10.6-universal/egg/mechanize/_urllib2_fork.py", line 571, in http_error_302 File "build/bdist.macosx-10.6-universal/egg/mechanize/_opener.py", line 188, in open File "build/bdist.macosx-10.6-universal/egg/mechanize/_mechanize.py", line 71, in http_request AttributeError: OpenerDirector instance has no attribute '_add_referer_header' </code></pre> <p>Any ideas?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload