Databases
StackOverflow2013
Postsdbo
POOutOfMemoryError while using HtmlUnit for scrapping
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOutOfMemoryError while using HtmlUnit for scrapping
    primarykey
    data
    text
    <p>I am using HtmlUnit to login on to a site and then download data from the table</p> <p>When I run my code is is causing <code>java.lang.OutOfMemoryError</code> And could not run further.</p> <p>Following is my code :</p> <pre><code>WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_6); webClient.getOptions().setJavaScriptEnabled(true); webClient.getOptions().setCssEnabled(false); webClient.getOptions().setRedirectEnabled(true); webClient.getCookieManager().setCookiesEnabled(true); webClient.getOptions().setPrintContentOnFailingStatusCode(false); webClient.setAjaxController(new NicelyResynchronizingAjaxController()); webClient.getOptions().setTimeout(50000); webClient.getOptions().setUseInsecureSSL(true); webClient.getOptions().setPopupBlockerEnabled(true); HtmlPage htmlPage=webClient.getPage(url); Thread.sleep(200); //~~~~~~~Log-In HtmlTextInput uname=(HtmlTextInput)htmlPage.getFirstByXPath("//*[@id=\"username\"]"); uname.setValueAttribute("xxx"); HtmlPasswordInput upass=(HtmlPasswordInput)htmlPage.getFirstByXPath("//*[@id=\"password\"]"); upass.setValueAttribute("xxx"); HtmlSubmitInput submit=(HtmlSubmitInput)htmlPage.getFirstByXPath("//*[@id=\"login-button\"]/input"); htmlPage=(HtmlPage) submit.click(); Thread.sleep(200); webClient.waitForBackgroundJavaScript(10000); for (int i = 0; i &lt; 250; i++) { if (!htmlPage.asText().contains("Loading...")) { break; } synchronized (htmlPage) { htmlPage.wait(500); } } System.out.println(htmlPage.asText()); </code></pre> <p>and Following is the stackTrace</p> <pre><code>java.lang.OutOfMemoryError: Java heap space at net.sourceforge.htmlunit.corejs.javascript.Node.newString(Node.java:155) at net.sourceforge.htmlunit.corejs.javascript.Node.newString(Node.java:151) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.createPropertyGet(IRFactory.java:1990) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformPropertyGet(IRFactory.java:968) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:106) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformPropertyGet(IRFactory.java:964) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:106) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformPropertyGet(IRFactory.java:964) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:106) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformFunctionCall(IRFactory.java:595) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:86) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformInfix(IRFactory.java:775) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:161) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformAssignment(IRFactory.java:368) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:152) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformExprStmt(IRFactory.java:488) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:149) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformBlock(IRFactory.java:406) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:82) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformIf(IRFactory.java:762) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:110) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformBlock(IRFactory.java:406) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:82) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformIf(IRFactory.java:762) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:110) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformBlock(IRFactory.java:406) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:82) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformIf(IRFactory.java:768) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:110) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformBlock(IRFactory.java:406) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transform(IRFactory.java:82) at net.sourceforge.htmlunit.corejs.javascript.IRFactory.transformFunction(IRFactory.java:560) </code></pre> <p>I have put follwoing lines in catlina.sh file to allot heap memory But still I am getting the same error (My RAM size is 2GB).</p> <pre><code>if [ -z "$LOGGING_MANAGER" ]; then JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" else JAVA_OPTS="$JAVA_OPTS $LOGGING_MANAGER" fi # Uncomment the following line to make the umask available when using the # org.apache.catalina.security.SecurityListener JAVA_OPTS="$JAVA_OPTS -Dorg.apache.catalina.security.SecurityListener.UMASK=`umask`" JAVA_OPTS="$JAVA_OPTS -Xms512m -Xmx2048m -XX:MaxPermSize=512m" JAVA_OPTS="-server -XX:+UseConcMarkSweepGC" </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USZero Piraeus
    1. USKunal Kishore
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POOutOfMemoryError while using HtmlUnit for scrapping
      1. USspeeday125
      1. VTBountyStart
    2. VO
      singulars
      1. POOutOfMemoryError while using HtmlUnit for scrapping
      1. USPSR
      1. VTFavorite
    3. VO
      singulars
      1. POOutOfMemoryError while using HtmlUnit for scrapping
      1. USspeeday125
      1. VTFavorite
    1. COLooking at the stack trace, you can see that it's recursing a lot. This doesn't seem to be the entire stack trace - how much did you truncate?
      singulars
      1. POOutOfMemoryError while using HtmlUnit for scrapping
      1. USAleks G
    2. COI have only put the cause from stack trace
      singulars
      1. POOutOfMemoryError while using HtmlUnit for scrapping
      1. USKunal Kishore
    3. COIn this stack trace there is no indication of your code, this is why I'm saying that you've truncated it. At some point there should be a line from your code (or a line indicating something like ...1234 more lines)
      singulars
      1. POOutOfMemoryError while using HtmlUnit for scrapping
      1. USAleks G
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload