StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHTML post in Java with HttpClient doesn't seem to work for a specific post
text
Body
copied!<p>I am using HttpClient by Apache to make one get call and two post calls to autozone.com. The get call is to access the homepage so that a JSessionId cookie is set. Then I am trying to search for stores around a certain zip code with their store locator and then set the first result as my "Preferred Store".</p> <p>I found out previously that all information about a user is stored server side and looked up using two permanent cookies. Basically just a username-password pair. These cookies are being created along with the JSessionId because the first post, which requires the JSessionId, is working perfectly. I also found out that they also don't directly return the results page, instead they return just Http headers with a 302 "Temporarily Moved" status. So I retrieve the new page from the headers and go there.</p> <p>This is exactly what I see happening with my program. The get call to the homepage is loaded successfully and the 3 necessary cookies are set and I retrieve the dynamic session confirmation number that is necessary for all post calls. Then I do the post call to the processing page for the StoreLocator passing all the necessary arguments, get the location of the results, do a get call and load that page. I find the "prefStoreNumber", the systems unique number to identify the store, and save it. Then I create my last post call. It goes to the correct page to process a preferred store selection. I again pass all the arguments that are in the form on the HTML page and the 302 response is returned. The problem arises when I get the new location. It should have, in the HTML, the store I just selected along the top bar, but instead it is still the same as if no store had ever been selected. I know that the cookies are being passed correctly or else I wouldn't have gotten the store results. </p> <p>I'm not sure why it isn't working so any ideas as to what might be happening are greatly appreciated. I've included my code in case I am just missing something that is necessary, but I have checked over it a thousand times and can not find anything wrong. And I used WireShark to see if the post was being created correctly and it was.</p> <pre><code> String useragent, homepage, locatorPage, host, sessionConf, results, preferedPage; homepage = "/autozone"; locatorPage = "/autozone/storelocator/storeLocatorMain.jsp?_DARGS=/autozone/storelocator/storeLocatorSearch.jsp"; preferedPage = "/autozone/storelocator/storeLocatorMain.jsp?_DARGS=/autozone/storelocator/storeLocatorResults.jsp"; useragent = "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1"; host = "http://www.autozone.com"; DefaultHttpClient client = new DefaultHttpClient(); client.getParams().setBooleanParameter("http.protocol.allow-circular-redirects", true); client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, useragent); HttpContext session = new BasicHttpContext(); HttpGet get = new HttpGet(host+homepage); get.setHeader("User-Agent", useragent); HttpEntity e = client.execute(get, session).getEntity(); sessionConf = getHTMLValue("_dynSessConf", EntityUtils.toString(e)); EntityUtils.consume(e); HttpPost post = new HttpPost(host+locatorPage); List<NameValuePair> form = new ArrayList<NameValuePair>(); form.add(new BasicNameValuePair("_dyncharset", "ISO-8859-1")); form.add(new BasicNameValuePair("_dynSessConf", sessionConf)); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.successURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=storeLocatorResults")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.successURL", " ")); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=null")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", " ")); form.add(new BasicNameValuePair("city", "")); form.add(new BasicNameValuePair("_D:city", " ")); form.add(new BasicNameValuePair("state", "-1")); form.add(new BasicNameValuePair("zip", Integer.toString(zip))); form.add(new BasicNameValuePair("_D:zip", " ")); form.add(new BasicNameValuePair("areaCode", "")); form.add(new BasicNameValuePair("_D:areaCode", " ")); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.findNearestLocations", "submit")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.findNearestLocations", " ")); form.add(new BasicNameValuePair("_DARGS", "/autozone/storelocator/storeLocatorSearch.jsp")); post.setEntity(new UrlEncodedFormEntity(form, "UTF-8")); HttpResponse response = client.execute(post, session); results = response.getFirstHeader("Location").toString().substring(10); EntityUtils.consume(response.getEntity()); get = new HttpGet(results); response = client.execute(get, session); e = response.getEntity(); String storeID = getHTMLValue("prefStoreNumber", EntityUtils.toString(e)); EntityUtils.consume(e); post = new HttpPost(host+preferedPage); form = new ArrayList<NameValuePair>(); form.add(new BasicNameValuePair("_dyncharset", "ISO-8859-1")); form.add(new BasicNameValuePair("_dynSessConf", sessionConf)); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.preferedStoreSuccessURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=storeLocatorResults")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.preferedStoreSuccessURL", " ")); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=storeLocatorResults")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", " ")); form.add(new BasicNameValuePair("prefStoreNumber", storeID)); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.setPreferedStore", "submit")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.setPreferedStore", " ")); form.add(new BasicNameValuePair("_DARGS", "/autozone/storelocator/storeLocatorResults.jsp")); post.setEntity(new UrlEncodedFormEntity(form, "UTF-8")); response = client.execute(post, session); displayResponse(response); results = response.getFirstHeader("Location").toString().substring(10); EntityUtils.consume(response.getEntity()); get = new HttpGet(results); response = client.execute(get, session); displayResponse(response); </code></pre> <p>The displayResponse is just for debugging. It lists all the response headers followed by a line separator and the main HTML. Sorry for the long-windedness but I wanted to make sure my problem was well understood. </p> <p>Update: Ok. I ran WireShark on a browser request and my program. I found out that I just missed a simple mistake. When I retrieved the prefStoreNumber from the HTML I accidently grabbed too much and had some random HTML stored with it. I am going to fix this and see if it works but there were also some other differences. I've listed them below. Would any of them give me trouble?:<br> Field: Browser|Program </p> <pre><code>Cache-Control: max-age=0|(N/A) Origin: http://www.autozone.com|(N/A) Content-Type: application/x-www-form-urlencoded|(N/A) Accept: text/html,application/xhtml+xml,application/x ml;q=0.9,*/*;q=0.8|(N/A) Referer: http://www.autozone .com/autozone/storelocator/storeLocatorMain.jsp? &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;targetPage=storeLocatorResults&_ requestid=3010533|(N/A) Accept-Encoding: gzip,deflate,sdch|(N/A) Accept-Language: en-US,en ;q=0.8|(N/A) Accept-Charset: ISO-8859 -1,utf-8;q=0.7,*;q=0.3|(N/A) Cookie2: (N/A)|$Version=1 </code></pre> <p>Update 2: It worked perfectly after I parsed the prefStoreNumber correctly.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload