Databases
StackOverflow2013
Postsdbo
POPeriodic HTML crawl
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPeriodic HTML crawl
    primarykey
    data
    text
    <p>I'm considering developing a site where the server will crawl another site periodically, in order to gather content for certain entries in my database. My quesitons are as follows...</p> <ol> <li>How do you get the server to execute a crawl timely?</li> <li>Can you get it to execute php or what language do you use to perform the crawl?</li> <li>Are there any good APIs to do this?</li> <li>Should I consider building my own? If so, some advice on how to get started would be great</li> </ol> <p>Basically, the kind of thing I want to do, is for the server to execute a script (say every hour), which finds all entries in the database which haven't yet been crawled on another site. It will take a certain value from those entries, and will use them to crawl another site... it might request a url like this: <code>www.anothersite.com/images?q=entryindb</code>.</p> <p>What I want it to do is then crawl the HTML, return an array, and log the values in the database. This is what I want the crawler to look for</p> <pre><code>Find all instances of &lt;img&gt; inside &lt;a&gt; inside &lt;td&gt; inside &lt;tr&gt; inside &lt;tbody&gt; inside &lt;table&gt; inside &lt;div id='content'&gt; Return array of the img.src from all instances. </code></pre> <p>Is something like that possible? - If so, how would I go about doing it? - Please bear in mind that web dev wise, the only experience I have so far (server-side) is with PHP.</p> <p><strong>UPDATE</strong>: I will be using a linux-based server, so I guess chron-scripting is how I should do it?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAlex Coplan
    1. USAlex Coplan
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPeriodic HTML crawl
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPeriodic HTML crawl
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPeriodic HTML crawl
      1. USShef
      1. VTFavorite
    1. CO@Robotsushi I scrape a specific bunch of scripts every day because the devs keep changing it at random times and I want to know when things break. There are entirely legit uses for this.
      singulars
      1. POPeriodic HTML crawl
      1. USIncognito
    2. COThis seems to be the perfect work for htmlagilitypack http://htmlagilitypack.codeplex.com/ but I realize that you want to implement this in PHP
      singulars
      1. POPeriodic HTML crawl
      1. USIcarus
    3. CO@Robotsushi - What an overly broad and fallacious statement. It *may be* "stealing" when you use a system in a way that is contrary to the terms of service. However, if a site makes data publicly available, scraping is no more stealing that simply viewing the web page in your browser. Or is your browser stealing when it makes a cache? How about Google's cache... stealing? Wayback machine, criminals? Don't generalize: it never works.
      singulars
      1. POPeriodic HTML crawl
      1. USChris Baker
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload