Databases
StackOverflow2013
Postsdbo
POWhat is a more efficient way of extracting a value from a webpage
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhat is a more efficient way of extracting a value from a webpage
    text
    copied!<p>I need to retrieve a value from a website (can vary and I have no control over the site). I currently have some code that works... but takes a very long time to run. I know that there is a vastly improved way of doing this, I just don't know what that is. </p> <p>I have considered several alternatives like Regex and the HTMLAgilityPack (seems complex and possibly overkill?) but without trying each of them I am not sure what would be most efficient. And I am sure there are many more possibilities as well. </p> <p>The problem may even be with how I am retrieving the page rather than how I am processing it.</p> <pre><code> Dim GETURL As WebRequest GETURL = WebRequest.Create("http://www.example.com") Dim objStream As Stream = GETURL.GetResponse.GetResponseStream() Dim objReader As New StreamReader(objStream) Dim sLine As String = "" Dim a As Integer = 0 Dim result As String = "" Do While Not sLine Is Nothing a += 1 sLine = objReader.ReadLine If Not sLine Is Nothing Then result += sLine End If Loop Dim startTag as string ="&lt;some html tag&gt;" Dim endTag as string ="&lt;closing tag&gt;" Dim firstIndex As Integer = result.IndexOf(startTag) + startTag.Length result = result.Substring(firstIndex, result.Length - firstIndex) Dim RequiredVal As String = result.Substring(0, result.IndexOf(endTag)) </code></pre> <p>Please note, I do realise just how hideously inefficient this code is, but rather than try loads of different permutations (and probably still have fairly inefficient code), I thought I would ask some experts for their advice first :-)</p> <p><strong>UPDATE:</strong></p> <p>As I didn't get any response (perhaps my question was a little too vague?) I have been trying to improve efficiency on my own. I have managed to decrease the time it takes to run by ~50% by using WebCient.DownloadString(). This is good but I suspect I can make improvements on extracting the data from the page. Please see updated code below:</p> <pre><code> Dim client As New WebClient() Dim result As String = client.DownloadString("http://www.example.com") Dim startTag as string ="&lt;some html tag&gt;" Dim endTag as string ="&lt;closing tag&gt;" Dim firstIndex As Integer = result.IndexOf(startTag) + startTag.Length result = result.Substring(firstIndex, result.Length - firstIndex) Dim RequiredVal As String = result.Substring(0, result.IndexOf(endTag)) </code></pre> <p>Any Suggestions would be greatly apprieciated.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload