StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHtmlUnit scraping google+ page javascript. Click show more button not working
text
Body
copied!<p><strong>i am trying to scrap this page <a href="https://plus.google.com/115016587855962294424/about" rel="nofollow">https://plus.google.com/115016587855962294424/about</a>. Everything works fine but when i try to click show more to load more reviews nothing happens here is my code</strong> </p> <pre><code>final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_24); page = webClient.getPage("https://plus.google.com/115016587855962294424/about"); assertEquals(200,page.getWebResponse().getStatusCode()); assertEquals("OK",page.getWebResponse().getStatusMessage()); System.out.println(page.getWebResponse().getStatusCode()); </code></pre> <p><strong>Clicking show more here</strong></p> <pre><code>HtmlSpan advancedSearchAn = (HtmlSpan) page.getFirstByXPath("//*[@id=\"115016587855962294424-about-page\"]/div/div[1]/div/div/div[2]/div[3]/span[1]"); page = advancedSearchAn.click(); </code></pre> <p><strong>but nothing happens i even tried</strong> </p> <pre><code>// webClient.waitForBackgroundJavaScript(10 * 1000); // webClient.setAjaxController(new NicelyResynchronizingAjaxController()); // webClient.setAjaxController(new AjaxController(){ // @Override // public boolean processSynchron(HtmlPage page, WebRequest request, boolean async) // { // return true; // } // }); </code></pre> <p><strong>Any suggestions ?</strong> </p> <p>UPDATE:</p> <p>*<strong><em>i was adviced to modify the incoming JavaScript code by subclass HttpWebConnection and override getResponse() as:</em>*</strong></p> <pre><code>new WebConnectionWrapper(webClient) { public WebResponse getResponse(WebRequest request) throws IOException { // System.out.println("content"); WebResponse response = super.getResponse(request); if (request.getUrl().toExternalForm().contains("https://plus.google.com/115016587855962294424/about")) { String content = response.getContentAsString("UTF-8"); //change content -- what is need to be changed System.out.println("content "+content); WebResponseData data = new WebResponseData(content.getBytes("UTF-8"), response.getStatusCode(), response.getStatusMessage(), response.getResponseHeaders()); response = new WebResponse(data, request, response.getLoadTime()); } System.out.println("content "+response.getContentAsString()); return response; } </code></pre> <p><em><strong>Any suggestions on how this can be done exactly and whats needed to be modified, i tried the following API's htmlunit jsoup webharvest selenium</em></strong> </p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload