Databases
StackOverflow2013
Postsdbo
POscraping an ASP website with php using cURL
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POscraping an ASP website with php using cURL
    text
    copied!<p>I am completely new to asp scraping and I need your help for the following code snippet:</p> <p>Aim of the code is to scrape the 2nd tab of a website that is retrieved via a <code>doPostBack</code> call. By using HTTPFOX I know the parameters passed in the POST Request. I want to reconstruct that POST request to make it work properly.</p> <p>Right now with <code>UrlEncode</code> I get a </p> <blockquote> <p>[FormatException: Invalid character in a Base-64 string.]</p> </blockquote> <p>error and when I don't send the right VIEWSTATE I get an invalid VIEWSTATE error.</p> <pre><code>function get_data_with_post_fields($url, $fields_data) { $ch = curl_init(); $timeout = 5; curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13'); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout); curl_setopt($ch, CURLOPT_ENCODING, ""); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS,$fields_data); curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1); $data = curl_exec($ch); curl_close($ch); return $data; } $VIEWSTATE = "/wEPDwUJNzU5NTE3MDU3..........moreeee........Qay9urEI="; $VIEWSTATE1 = urlencode( $VIEWSTATE ) ; $url_to_scrape = 'http://www.awebsite.com/en/product/393633343239/ProductDetails.aspx'; $PF = array('__VIEWSTATE' =&gt; $VIEWSTATE1, '__EVENTVALIDATION' =&gt; "", '__EVENTARGUMENT' =&gt; "", '__EVENTTARGET' =&gt; urlencode("ctl00$cphMain$lbTab2"), ); print $raw = get_data_with_post_fields($url_to_scrape, $PF); </code></pre> <p>How can I reconstruct my POST request?</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload