Databases
StackOverflow2013
Postsdbo
POHTML post in Java with HttpClient doesn't seem to work for a specific post
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHTML post in Java with HttpClient doesn't seem to work for a specific post
    primarykey
    data
    text
    <p>I am using HttpClient by Apache to make one get call and two post calls to autozone.com. The get call is to access the homepage so that a JSessionId cookie is set. Then I am trying to search for stores around a certain zip code with their store locator and then set the first result as my "Preferred Store".</p> <p>I found out previously that all information about a user is stored server side and looked up using two permanent cookies. Basically just a username-password pair. These cookies are being created along with the JSessionId because the first post, which requires the JSessionId, is working perfectly. I also found out that they also don't directly return the results page, instead they return just Http headers with a 302 "Temporarily Moved" status. So I retrieve the new page from the headers and go there.</p> <p>This is exactly what I see happening with my program. The get call to the homepage is loaded successfully and the 3 necessary cookies are set and I retrieve the dynamic session confirmation number that is necessary for all post calls. Then I do the post call to the processing page for the StoreLocator passing all the necessary arguments, get the location of the results, do a get call and load that page. I find the "prefStoreNumber", the systems unique number to identify the store, and save it. Then I create my last post call. It goes to the correct page to process a preferred store selection. I again pass all the arguments that are in the form on the HTML page and the 302 response is returned. The problem arises when I get the new location. It should have, in the HTML, the store I just selected along the top bar, but instead it is still the same as if no store had ever been selected. I know that the cookies are being passed correctly or else I wouldn't have gotten the store results. </p> <p>I'm not sure why it isn't working so any ideas as to what might be happening are greatly appreciated. I've included my code in case I am just missing something that is necessary, but I have checked over it a thousand times and can not find anything wrong. And I used WireShark to see if the post was being created correctly and it was.</p> <pre><code> String useragent, homepage, locatorPage, host, sessionConf, results, preferedPage; homepage = "/autozone"; locatorPage = "/autozone/storelocator/storeLocatorMain.jsp?_DARGS=/autozone/storelocator/storeLocatorSearch.jsp"; preferedPage = "/autozone/storelocator/storeLocatorMain.jsp?_DARGS=/autozone/storelocator/storeLocatorResults.jsp"; useragent = "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1"; host = "http://www.autozone.com"; DefaultHttpClient client = new DefaultHttpClient(); client.getParams().setBooleanParameter("http.protocol.allow-circular-redirects", true); client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, useragent); HttpContext session = new BasicHttpContext(); HttpGet get = new HttpGet(host+homepage); get.setHeader("User-Agent", useragent); HttpEntity e = client.execute(get, session).getEntity(); sessionConf = getHTMLValue("_dynSessConf", EntityUtils.toString(e)); EntityUtils.consume(e); HttpPost post = new HttpPost(host+locatorPage); List&lt;NameValuePair&gt; form = new ArrayList&lt;NameValuePair&gt;(); form.add(new BasicNameValuePair("_dyncharset", "ISO-8859-1")); form.add(new BasicNameValuePair("_dynSessConf", sessionConf)); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.successURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=storeLocatorResults")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.successURL", " ")); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=null")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", " ")); form.add(new BasicNameValuePair("city", "")); form.add(new BasicNameValuePair("_D:city", " ")); form.add(new BasicNameValuePair("state", "-1")); form.add(new BasicNameValuePair("zip", Integer.toString(zip))); form.add(new BasicNameValuePair("_D:zip", " ")); form.add(new BasicNameValuePair("areaCode", "")); form.add(new BasicNameValuePair("_D:areaCode", " ")); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.findNearestLocations", "submit")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.findNearestLocations", " ")); form.add(new BasicNameValuePair("_DARGS", "/autozone/storelocator/storeLocatorSearch.jsp")); post.setEntity(new UrlEncodedFormEntity(form, "UTF-8")); HttpResponse response = client.execute(post, session); results = response.getFirstHeader("Location").toString().substring(10); EntityUtils.consume(response.getEntity()); get = new HttpGet(results); response = client.execute(get, session); e = response.getEntity(); String storeID = getHTMLValue("prefStoreNumber", EntityUtils.toString(e)); EntityUtils.consume(e); post = new HttpPost(host+preferedPage); form = new ArrayList&lt;NameValuePair&gt;(); form.add(new BasicNameValuePair("_dyncharset", "ISO-8859-1")); form.add(new BasicNameValuePair("_dynSessConf", sessionConf)); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.preferedStoreSuccessURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=storeLocatorResults")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.preferedStoreSuccessURL", " ")); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", "/autozone/storelocator/storeLocatorMain.jsp?targetPage=storeLocatorResults")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.errorURL", " ")); form.add(new BasicNameValuePair("prefStoreNumber", storeID)); form.add(new BasicNameValuePair("/autozone/diy/storelocator/StoreLocatorFormHandler.setPreferedStore", "submit")); form.add(new BasicNameValuePair("_D:/autozone/diy/storelocator/StoreLocatorFormHandler.setPreferedStore", " ")); form.add(new BasicNameValuePair("_DARGS", "/autozone/storelocator/storeLocatorResults.jsp")); post.setEntity(new UrlEncodedFormEntity(form, "UTF-8")); response = client.execute(post, session); displayResponse(response); results = response.getFirstHeader("Location").toString().substring(10); EntityUtils.consume(response.getEntity()); get = new HttpGet(results); response = client.execute(get, session); displayResponse(response); </code></pre> <p>The displayResponse is just for debugging. It lists all the response headers followed by a line separator and the main HTML. Sorry for the long-windedness but I wanted to make sure my problem was well understood. </p> <p>Update: Ok. I ran WireShark on a browser request and my program. I found out that I just missed a simple mistake. When I retrieved the prefStoreNumber from the HTML I accidently grabbed too much and had some random HTML stored with it. I am going to fix this and see if it works but there were also some other differences. I've listed them below. Would any of them give me trouble?:<br> Field: Browser|Program </p> <pre><code>Cache-Control: max-age=0|(N/A) Origin: http://www.autozone.com|(N/A) Content-Type: application/x-www-form-urlencoded|(N/A) Accept: text/html,application/xhtml+xml,application/x ml;q=0.9,*/*;q=0.8|(N/A) Referer: http://www.autozone .com/autozone/storelocator/storeLocatorMain.jsp? &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;targetPage=storeLocatorResults&amp;_ requestid=3010533|(N/A) Accept-Encoding: gzip,deflate,sdch|(N/A) Accept-Language: en-US,en ;q=0.8|(N/A) Accept-Charset: ISO-8859 -1,utf-8;q=0.7,*;q=0.3|(N/A) Cookie2: (N/A)|$Version=1 </code></pre> <p>Update 2: It worked perfectly after I parsed the prefStoreNumber correctly.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJaws212
    1. USJaws212
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. COYou say that your requests are being sent correctly--based on what? Are you going from some documentation? Did you try it with a browser? If you're trying to duplicate browser behavior, have you compared your HttpClient request with one sent from a browser? If requests from a browser work one way and requests from HttpClient work another, then the requests are different.
      singulars
      1. POHTML post in Java with HttpClient doesn't seem to work for a specific post
      1. USRyan Stewart
    2. COI know my requests are correct because the page returned is the same page returned as when I use a browser to do the action and if they failed I wouldn't be able to get the "prefStore" number. I used WireShark to compare the requests to normal browser behavior and as far as I remember they are identical. Although the "Accept" header may have been slightly different but I didn't think that would have an impact. I will run the test again when I get home and post the results. The only other thing that is different are the way the responses are sent.
      singulars
      1. POHTML post in Java with HttpClient doesn't seem to work for a specific post
      1. USJaws212
    3. COSo when you get a chance, check the request you're having trouble with--the final post after getting the prefStoreNumber--to see if there's any difference between a browser and HttpClient request. One thing to keep an eye out for: user agent strings. A lot of places check the UA string to verify that it looks like a browser making the request and will respond differently if not.
      singulars
      1. POHTML post in Java with HttpClient doesn't seem to work for a specific post
      1. USRyan Stewart
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload