Databases
StackOverflow2013
Postsdbo
PODownload a cookie to make new GET request
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PODownload a cookie to make new GET request
    primarykey
    data
    text
    <p>I am trying to do a PHP GET request to a website:</p> <p><img src="https://i.stack.imgur.com/oE9eZ.jpg" alt="enter image description here"></p> <p>The problem is that this website will only process my request if I attach Cookie information to the header of the request.</p> <p>Or in picture terms, if I disable cookies in my browser, I get this:</p> <p><img src="https://i.stack.imgur.com/ZOCsn.jpg" alt="enter image description here"></p> <p>Which means the website recognises that it's my first time 'visiting' the site.</p> <p>Problem is, that if I now use the search bar on the top right, it will <strong>not</strong> process this request: it will just show the same (general) screen.</p> <p>E.g.: if I have cookies <em>disabled</em> and I search for "AAPL", it will not show any results.</p> <p>Now if I have cookies <em>enabled</em>, the request is handled just fine:</p> <p><img src="https://i.stack.imgur.com/PVFSX.jpg" alt="enter image description here"></p> <p>And so the "AAPL" results are shown.</p> <p>You can try this yourself as well:</p> <p>With cookies <em>enabled</em>, visit <code>http://www.pennystocktweets.com/user_posts/feeds?cat=search&amp;lptyp=prep&amp;usrstk=AAPL</code></p> <p>With cookies <em>disabled</em>, visit the link again: <code>http://www.pennystocktweets.com/user_posts/feeds?cat=search&amp;lptyp=prep&amp;usrstk=AAPL</code></p> <p>Now compare the responses, only the first one is correct.</p> <p>This means that the website only works <strong>after</strong> the client has downloaded a cookie, and then has made another (new) GET request to the server with this Cookie information attached.</p> <p>(Does this imply that the website <strong>needs</strong> a session-cookie to function correctly?)</p> <p>Now what I'm trying to do is imitate the request with <a href="http://hc.apache.org/httpclient-3.x/" rel="nofollow noreferrer">Apache HttpClient</a> like so:</p> <pre><code>import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.io.UnsupportedEncodingException; import java.net.CookieHandler; import java.net.CookieManager; import java.net.HttpURLConnection; import java.net.URL; import java.util.Date; import java.util.List; import java.util.StringTokenizer; import org.apache.http.HttpResponse; import org.apache.http.NameValuePair; import org.apache.http.client.HttpClient; import org.apache.http.client.entity.UrlEncodedFormEntity; import org.apache.http.client.methods.HttpGet; import org.apache.http.client.methods.HttpPost; import org.apache.http.impl.client.DefaultHttpClient; import org.apache.http.message.BasicNameValuePair; public class downloadTweets { private String cookies; private HttpClient client = new DefaultHttpClient(); private final String USER_AGENT = "Mozilla/5.0"; public static void main(String[] args) throws Exception { String ticker = "AAPL"; String lptyp = "prep"; int opid = 0; int lpid = 0; downloadTweets test = new downloadTweets(); String url = test.constructURL(ticker, lptyp, opid, lpid); // make sure cookies is turn on CookieHandler.setDefault(new CookieManager()); downloadTweets http = new downloadTweets(); String page = http.GetPageContent(url, ticker); System.out.println(page); } public String constructURL(String ticker, String lptyp, int opid, int lpid) { String link = "http://www.pennystocktweets.com/user_posts/feeds?cat=search" + "&amp;lptyp=" + lptyp + "&amp;usrstk=" + ticker; if (opid != 0) { link = link + "&amp;opid=" + opid + "&amp;lpid=" + lpid; } return link; } private String GetPageContent(String url, String ticker) throws Exception { HttpGet request = new HttpGet(url); String RefererLink = "http://www.pennystocktweets.com/search/post/" + ticker.toUpperCase(); request.setHeader("Host", "www.pennystocktweets.com"); request.setHeader("Connection", "Keep-alive"); request.setHeader("Accept", "*/*"); request.setHeader("X-Requested-With", "XMLHttpRequest"); request.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36"); request.setHeader("Referer", RefererLink); request.setHeader("Accept-Language", "nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4,fr;q=0.2"); HttpResponse response = client.execute(request); int responseCode = response.getStatusLine().getStatusCode(); System.out.println("\nSending 'GET' request to URL : " + url); System.out.println("Response Code : " + responseCode); BufferedReader rd = new BufferedReader( new InputStreamReader(response.getEntity().getContent())); StringBuffer result = new StringBuffer(); String line = ""; while ((line = rd.readLine()) != null) { result.append(line); } // set cookies setCookies(response.getFirstHeader("Set-Cookie") == null ? "" : response.getFirstHeader("Set-Cookie").toString()); return result.toString(); } public String getCookies() { return cookies; } public void setCookies(String cookies) { this.cookies = cookies; } } </code></pre> <p>Now, the same thing holds: if I attach (my) cookie information, the response works just fine, and if I don't the response doesn't work.</p> <p><strong>But I don't know how to get the cookie information and then use it in a new GET request.</strong></p> <p>So my question is:</p> <p>How can I make 2 requests to a website such that:</p> <p><strong>On the first GET request</strong>, I get cookie information from the website and store this in my Java program</p> <p><strong>On the second GET request</strong>, I use the stored cookie information (as a Header) to make a new request.</p> <p><strong>Note:</strong> <em>I don't know if the cookie is a normal cookie or a session cookie but I suspect it's a session-cookie</em>!</p> <p>All help is greatly appreciated!</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJean-Paul
    1. USJean-Paul
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload