StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PONot able to scrape a website for HTML?
text
Body
copied!<p>So I'm trying to get my application to go to website, get the HTML from that website, remove the unnecessary elements from the HTML, and then load the 'content' in my make shift app, because I don't have an API or a feed. I'm using Jsoup and it is able to work if I'm doing the web scraping not in android, but android is not liking it.</p> <pre><code>public class SimpleDiggActivity extends Activity { private WebView browser; final Activity activity = this; @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); this.getWindow().requestFeature(Window.FEATURE_PROGRESS); setContentView(R.layout.main); getWindow().setFeatureInt(Window.FEATURE_PROGRESS, Window.PROGRESS_VISIBILITY_ON); String url = "http://www.digg.com"; Document digg; browser = (WebView) findViewById(R.id.mybrowser); final Button homeDigg = (Button) findViewById(R.id.button1); browser.setWebViewClient(new SimpleWebViewClient()); browser.getSettings().setJavaScriptEnabled(true); browser.getSettings().setUseWideViewPort(true); browser.getSettings().setLoadWithOverviewMode(true); browser.getSettings().setDisplayZoomControls(false); browser.getSettings().setEnableSmoothTransition(true); browser.getSettings().setBuiltInZoomControls(true); browser.getSettings().setUserAgentString("Android"); // progressCircle = ProgressDialog.show(SimpleDiggActivity.this, "", "Loading..."); final ProgressDialog progressCircle = new ProgressDialog(activity); progressCircle.setProgressStyle(ProgressDialog.STYLE_SPINNER); progressCircle.setMessage("Loading..."); progressCircle.setCancelable(false); try{ Toast.makeText(getApplicationContext(), "No Steps down", Toast.LENGTH_SHORT).show(); Document diggTest = Jsoup.connect("http://digg.com/enable/mobile").get(); Toast.makeText(getApplicationContext(), "1 Steps down", Toast.LENGTH_SHORT).show(); String diggTitle = diggTest.title(); Toast.makeText(getApplicationContext(), "2 Steps down" , Toast.LENGTH_SHORT).show(); Document compressed = Jsoup.parseBodyFragment(diggTitle); Toast.makeText(getApplicationContext(), "3 Steps down", Toast.LENGTH_SHORT).show(); org.jsoup.select.Elements div = diggTest.select("div"); Toast.makeText(getApplicationContext(), "4 Steps down", Toast.LENGTH_SHORT).show(); String divBrow = div.toString(); Toast.makeText(getApplicationContext(), "5 Steps down", Toast.LENGTH_SHORT).show(); browser.loadUrl(divBrow); }catch (Exception e){ e.printStackTrace(); Toast.makeText(getApplicationContext(), "Gave up", Toast.LENGTH_SHORT).show(); String diggBrow = url; browser.loadUrl("http://www.google.com"); } </code></pre> <p>Sorry if it's messy, I'm just messing around and it's my first time. The Toasts are for me to tell when the code is failing the try and resorting to the catch. When I run it, it doesn't make it past</p> <pre><code> Document diggTest = Jsoup.connect("http://digg.com/enable/mobile").get(); </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload