Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>You need to use a library that has javascript support. I use <strong>HtmlUnit</strong> for this which is a great library for replicating browser behavior!</p> <p>See my modified answer <a href="https://stackoverflow.com/questions/14740801/easiest-way-to-browse-to-a-page-and-submit-form-in-java/14905070#14905070" title="from this question">from this question</a> below for a simple example of how to access a page with javascript. </p> <p><strong>First</strong>, check out their web page(<a href="http://htmlunit.sourceforge.net/" rel="nofollow noreferrer">http://htmlunit.sourceforge.net/</a>) to get htmlunit up and running. Make sure you use the latest snapshot(2.12 when writing this)</p> <p>Try these settings to ignore pretty much any obstacle:</p> <pre><code>WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17); webClient.getOptions().setRedirectEnabled(true); webClient.getOptions().setCssEnabled(false); webClient.getOptions().setThrowExceptionOnScriptError(false); webClient.getOptions().setThrowExceptionOnFailingStatusCode(false); webClient.getOptions().setUseInsecureSSL(true); webClient.getOptions().setJavaScriptEnabled(true); webClient.getCookieManager().setCookiesEnabled(true); </code></pre> <p>Then when fetching your page, make sure you wait for background Javascript before doing anything with the page, like waiting for background javascript.</p> <pre><code>//Get Page HtmlPage page1 = webClient.getPage("https://login-url/"); //Wait for background Javascript webClient.waitForBackgroundJavaScript(10000); //Get full page _after_ javascript has rendered it fully System.out.println(page1.asXml()); </code></pre> <p>I hope this basic example will help you!</p> <p>You can use HtmlUnit to do pretty much anything a browser can do, but programmatically.</p>
    singulars
    1. This table or related slice is empty.
    1. POScraping data from webpage using Java?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USCommunity
    1. USOakNinja
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POScraping data from webpage using Java?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COthanks this is interesting stuff. I just realized there's another layer of complexity. The page that i linked to above does not include the twitter handle name. I have to individually click on each link and then a small box is displayed using javascript that includes the twitter handle (@name). will htmlPage handle this. I guess i have to print the entire page everytime i click on a link.
      singulars
      1. PO
      1. USAli
    2. COactually i just checked the sample webpage and the links display the twitter name, not the twitter handle (@name), which links to www.twitter.com/@name in the xml file i get from htmlUnit will it include <a href="www.twitter.com/@name">Twitter Name</a>
      singulars
      1. PO
      1. USAli
    3. COYou can do that with htmlunit. If the links are hidden divs with already loaded data it should be visible in the source code, no need to get htmlunit to click them. If you'd like to interact with the page, see my original answer in the linked question in the beginning of my answer!
      singulars
      1. PO
      1. USOakNinja
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload