Databases
StackOverflow2013
Postsdbo
POScraping countermeasures on nginx?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POScraping countermeasures on nginx?
    primarykey
    data
    text
    <p>I'm writing some code that has to get some data from a website. Nothing controversial (I think) - it's for a kid's sports club, and it has to get their times from the national organisation's website. It's not proprietary or commercial data.</p> <p>The problem is that the returned data appears to be deliberately corrupted. I may just be being paranoid, but I've spent a few hours checking this. I'm using my own code, and I'm using the live headers Firefox extension to find out what to send to the site. I'm duplicating the <code>GET</code> headers exactly, except that I'm leaving out <code>Accept-Encoding</code>, since I don't want to handle gzip. I've tried <code>Connection</code> set to both <code>close</code> and <code>keep-alive</code>, but it makes no difference.</p> <p>The returned page has a few additional hex character sequences spread around it - nothing much, but it's enough to mess up my parsing. The characters and their location change every time I try this. My initial thought was that I was messing up the stitching together of the buffers I was getting back (I have to call <code>recv</code> maybe 20 times to get the entire page), but this doesn't seem to be the problem. The scraped version of the page always ends like this, for example:</p> <pre><code>&lt;/body&gt; 7 &lt;/html&gt; 0 </code></pre> <p>where the live page always ends <code>&lt;/body&gt;&lt;/html&gt;</code>.</p> <p>Any idea what's going on here? This site appears to be on Cloudflare/nginx. Is this something that nginx can do? Is it possible that they're messing up the text version of the page, and sending good data on the gzipped version? I'm not keen to start unzipping data.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USEML
    1. USEML
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COI seem to remember running in to something similar a while back, and figuring out a solution. I don't recall offhand what it was, though. Could you post the exact URL you're trying to request?
      singulars
      1. POScraping countermeasures on nginx?
      1. UStodd
    2. COSure - go to http://www.swimmingresults.org/individualbest/, type in an arbitrary surname, and you're there. A web search on `7</html>0` brings up copies of a couple of nginx web pages, which is an interesting data point. Thanks.
      singulars
      1. POScraping countermeasures on nginx?
      1. USEML
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload