Databases
StackOverflow2013
Postsdbo
POIs it better to try for one mega screen scraper or split it into a scraper for different sites?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs it better to try for one mega screen scraper or split it into a scraper for different sites?
    primarykey
    data
    text
    <p>I will explain my situation.</p> <p>Our Social Media Manager (yay) suddenly wants something to scrape a list of about 40 websites for information about our company, for example there's a lot of review sites in the list.</p> <p>(I have read a ton of tutorials and SO questions but still) My questions are:</p> <ol> <li><p>Is it possible to build a generic scraper that will work across all of these sites or do I need a separate scraper for each site?</p></li> <li><p>I think I understand how to parse an individual web page but how do you do it, where, for example there's a website structure of <code>review-website.com/company-name</code> and on that page are titles and a snippet of the review that then link to the actual full page review?</p></li> </ol> <p>i.e. Crawling and scraping multiple pages on multiple sites. Some are 'easier' than others because they have dedicated pages like the urls previously mentioned but some are forums etc with no particular structure that just happen to mention our company name so I don't know how to get relevant information on those.</p> <ol> <li>Does the time spent creating this justify that the Social Media Manager could just search these sites manually himself? Especially considering that a HTML change on any of the sites could possibly end up breaking the scraper?</li> </ol> <p>I really don't think this is a good idea yet my Line Manager seems to think it will take a morning's worth of work to write a scraper for all of these sites and I have no idea how to do it!</p> <p><strong>UPDATE</strong></p> <p>Thank you very much for the answers so far, I also thought I'd provide a list of the sites just to clarify what I think is an extreme task:</p> <p><code>Facebook - www.facebook.com</code></p> <p><code>Social Mention - www.socialmention.com</code></p> <p><code>Youtube - www.youtube.com</code></p> <p><code>Qype - www.qype.co.uk</code></p> <p><code>Money Saving Expert - www.moneysavingexpert.co.uk</code></p> <p><code>Review Centre - www.reviewcentre.com</code></p> <p><code>Dooyoo - www.dooyoo.co.uk</code></p> <p><code>Yelp - www.yelp.co.uk</code></p> <p><code>Ciao - www.ciao.co.uk</code></p> <p><code>All in London - www.allinlondon.co.uk</code></p> <p><code>Touch Local - www.touchlocal.com</code></p> <p><code>Tipped - www.tipped.co.uk</code></p> <p><code>What Clinic - www.whatclinic.com</code></p> <p><code>Wahanda - www.wahanda.com</code></p> <p><code>Up My Street - www.upmystreet.com</code></p> <p><code>Lasik Eyes - www.lasik-eyes.co.uk/</code></p> <p><code>Lasik Eyes (Forum) - forums.lasik-eyes.co.uk/default.asp</code></p> <p><code>Laser Eye Surgery - www.laser-eye-surgery-review.com/</code></p> <p><code>Treatment Saver - www.treatmentsaver.com/lasereyesurgery</code></p> <p><code>Eye Surgery Compare - www.eyesurgerycompare.co.uk/best-uk-laser-eye-surgery-clinics</code></p> <p><code>The Good Surgeon Guide - www.thegoodsurgeonguide.co.uk/</code></p> <p><code>Private Health -www.privatehealth.co.uk/hospitaltreatment/find-a-treatment/laser-eye-surgery/</code></p> <p><code>Laser Eye Surgery Wiki - www.lasereyesurgerywiki.co.uk</code></p> <p><code>PC Advisor - www.pcadvisor.co.uk/forums/2/consumerwatch/</code></p> <p><code>Scoot - www.scoot.co.uk</code></p> <p><code>Cosmetic Surgery Reviews - www.cosmetic-surgery-reviews.co.uk</code></p> <p><code>Lasik Reviews - www.lasikreviews.co.uk</code></p> <p><code>Laser Eye Surgery Costs - www.lasereyesurgerycosts.co.uk</code></p> <p><code>Who Calls Me - www.whocallsme.com</code></p> <p><code>Treatment Adviser - www.treatmentadviser.com/</code></p> <p><code>Complaints Board - http://www.complaintsboard.com</code></p> <p><code>Toluna - http://uk.toluna.com/</code></p> <p><code>Mums Net - http://www.mumsnet.com</code></p> <p><code>Boards.ie - http://www.boards.ie</code></p> <p><code>AV Forums - http://www.avforums.com</code></p> <p><code>Magic Mum - http://www.magicmum.com</code></p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USmartincarlin87
    1. USmartincarlin87
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POIs it better to try for one mega screen scraper or split it into a scraper for different sites?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POIs it better to try for one mega screen scraper or split it into a scraper for different sites?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POIs it better to try for one mega screen scraper or split it into a scraper for different sites?
      1. USSwair
      1. VTFavorite
    1. COBtw, you may want to point your manager towards the company Legal department, to check whether you're even allowed to scrape those sites.
      singulars
      1. POIs it better to try for one mega screen scraper or split it into a scraper for different sites?
      1. USPiskvor
    2. COA little thing that is often overlooked by people making scrapers: Be sure to respect `robots.txt`. It's not there just because of the cool filename. Keep in mind that the presence of a robots.txt doesn't mean you may copy everything that isn't mentioned in it. It's a blacklist, nothing more.
      singulars
      1. POIs it better to try for one mega screen scraper or split it into a scraper for different sites?
      1. USRikkusRukkus
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload