Databases
StackOverflow2013
Postsdbo
POUse cURL or wget with http POST to reach search results after the first page
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUse cURL or wget with http POST to reach search results after the first page
    text
    copied!<p>EDIT: I've got a much more specific idea of what I'm looking for now so I'm re-writing the whole question.</p> <p>My overall goal is to get to the search results after the first page (from within a script) on the webpage <a href="http://www.ncbi.nlm.nih.gov/images" rel="nofollow">http://www.ncbi.nlm.nih.gov/images</a>. Using the Firefox extension "Tamper Data", I have inspected the requests sent by my browser and found that I am able to modify the http POST request to get to any page of the results.</p> <p>Now I would like to do this within a script. I've tried both </p> <pre><code>wget --post-data 'var1=foo&amp;var2=bar&amp;var3=...' http://www.ncbi.nlm.nih.gov/images </code></pre> <p>and</p> <pre><code>cURL --data 'var1=foo&amp;var2=bar&amp;var3=...' http://www.ncbi.nlm.nih.gov/images </code></pre> <p>and I've tried making the initial request to <a href="http://www.ncbi.nlm.nih.gov/images?term=INSERTSEARCHTERMHERE" rel="nofollow">http://www.ncbi.nlm.nih.gov/images?term=INSERTSEARCHTERMHERE</a> and saving a cookie, then loading the cookie the next time I request, this time with POST data indicating page number. It doesn't work. Anytime I request to the first URL I get the home page for image search or I get a page titled "Images - Error encountered" with no search results. If I request to the second URL (replacing INSERTSEARCHTERMHERE with my actual search term) I always get the first page of the results, even though I sent POST data including a variable asking for a different page. It seems there are two - maybe three? - variables denoting page number: </p> <pre><code>EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_Pager.cPage=14 EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_Pager.CurrPage=14 </code></pre> <p>and in Tamper Data this is always the current page (the one I was on when I made the request for a new page):</p> <pre><code>EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_Pager.cPage=1 </code></pre> <p>(Yes, there are two variables in the POST data with the same name - I don't know what that is about...??)</p> <p>So how can I use cURL or wget within a script to get to all of the pages of the search results? Thanks for your help! (and thanks to the commenters for helping me clarify the question!)</p> <p>Additional info: There are a ton of POST fields, and I am sending all of them. I copied this out of what Tamper Data recorded:</p> <p>EntrezSystem2.PEntrez.ImagesDb.Images_SearchBar.SearchResourceList=images&amp;EntrezSystem2.PEntrez.ImagesDb.Images_SearchBar.Term=drug&amp;EntrezSystem2.PEntrez.ImagesDb.Images_SearchBar.CurrDb=images&amp;EntrezSystem2.PEntrez.ImagesDb.Entrez_PageController.PreviousPageName=results&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.sPresentation=docsum&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.sPageSize=20&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.FileFormat=docsum&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.LastPresentation=docsum&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.Presentation=docsum&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.PageSize=20&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.LastPageSize=20&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.Format=&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.LastFormat=&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_Pager.cPage=14&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_Pager.CurrPage=14&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_ResultsController.ResultCount=38231&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_ResultsController.RunLastQuery=&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_Pager.cPage=1&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.sPresentation2=docsum&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Images_DisplayBar.sPageSize2=20&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_MultiItemSupl.Discovery_SearchDetails.SearchDetailsTerm=drug%5BAll+Fields%5D&amp;EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.HistoryDisplay.Cmd=PageChanged&amp;EntrezSystem2.PEntrez.DbConnector.Db=images&amp;EntrezSystem2.PEntrez.DbConnector.LastDb=images&amp;EntrezSystem2.PEntrez.DbConnector.Term=drug&amp;EntrezSystem2.PEntrez.DbConnector.LastTabCmd=&amp;EntrezSystem2.PEntrez.DbConnector.LastQueryKey=1&amp;EntrezSystem2.PEntrez.DbConnector.IdsFromResult=&amp;EntrezSystem2.PEntrez.DbConnector.LastIdsFromResult=&amp;EntrezSystem2.PEntrez.DbConnector.LinkName=&amp;EntrezSystem2.PEntrez.DbConnector.LinkReadableName=&amp;EntrezSystem2.PEntrez.DbConnector.LinkSrcDb=&amp;EntrezSystem2.PEntrez.DbConnector.Cmd=PageChanged&amp;EntrezSystem2.PEntrez.DbConnector.TabCmd=&amp;EntrezSystem2.PEntrez.DbConnector.QueryKey=&amp;p%24a=EntrezSystem2.PEntrez.ImagesDb.Images_ResultsPanel.Entrez_Pager.cPage&amp;p%24l=EntrezSystem2&amp;p%24st=images</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload