Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I am currently writing a web extraction framework. I have 524 tests that get data from 250 websites using XPath. Initially the framework used a HTML parser, HTMLCleaner, but I am currently investigating using <a href="http://docs.seleniumhq.org/" rel="noreferrer">Selenium</a> because I want Javascript support. I have run the tests against the HtmlUnit, Chrome, Firefox and <a href="https://github.com/detro/ghostdriver" rel="noreferrer">PhantomJS drivers</a>. Here is a comparison of the time taken and the number of failures for each approach:</p> <pre><code> Failures Time (secs) HtmlCleaner 0 82 HtmlUnit 169 102 Google Chrome 38 562 Firefox 46 1159 PhantomJS 40 575 </code></pre> <p>Some comments:</p> <ul> <li><p>In some cases the "failures" may not be failures at all, it may be that the extractors are failing because Javascript is re-writing the DOM. I am in the process of analyzing the failures to find the cause.</p></li> <li><p>That said, HtmlUnit is the fastest Selenium driver but it is also unreliable. This unreliability does not just concern Javascript, there are problems processing "messy, dirty, real-world" HTML because something seems to be broken in the tag balancing algorithm. A couple of issues have been raised about this but they have not been fixed - see <a href="https://sourceforge.net/p/htmlunit/bugs/1423/" rel="noreferrer">HTML-UNIT 1423</a> and <a href="https://sourceforge.net/p/htmlunit/bugs/1046/" rel="noreferrer">HTML-UNIT 1046</a>. </p></li> <li><p>Firefox is the slowest Selenium driver, even though I am disabling image loading and stylesheets. This is because it is the slowest to load and initialize, making it considerably slower than Chrome, and every time an extraction fails I need to reload the driver (in the tests I create a pool of 5 drivers to mitigate the URL retrieval delays for all the Selenium web drivers).</p></li> <li><p>PhantomJS achieves a better accuracy than Firefox, slightly lower than Chrome, but in around half the time of Firefox. What is more, I can run it on my dev box, it does not "take over my machine" by launching multiple browsers so I can get on with work.</p></li> </ul> <p>I would highly recommend PhantomJS.</p>
    singulars
    1. This table or related slice is empty.
    1. POCasperjs/PhantomJs vs Selenium
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USMark Butler
    1. USMark Butler
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIt takes 9 minutes to run your test suite with phantomJS? That must feel like forever...
      singulars
      1. PO
      1. USKevin
    2. CO@Kevin Yes :) - but HTMLCleaner is the standard test, I mark use [JUnit categories](http://stackoverflow.com/questions/15418466/how-to-include-multiple-categories-in-junit4) to mark the other tests as optional so they aren't part of the standard unit tests
      singulars
      1. PO
      1. USMark Butler
    3. COThanks. Yeah I've been working on full-js tests and they're so glacial - like 15-20 seconds for a few page test. Perhaps my scale of 'glacial' needs to be adjusted though hehe :) Odd though that if I do it manually, it only takes ~5 seconds to click through the forms.
      singulars
      1. PO
      1. USKevin
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload