Databases
StackOverflow2013
Postsdbo
POHTML Agility Pack Screen Scraping XPATH isn't returning data
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHTML Agility Pack Screen Scraping XPATH isn't returning data
    text
    copied!<p>I'm attempting to write a screen scraper for Digikey that will allow our company to keep accurate track of pricing, part availability and product replacements when a part is discontinued. There seems to be a discrepancy between the XPATH that I'm seeing in Chrome Devtools as well as Firebug on Firefox and what my C# program is seeing.</p> <p>The page that I'm scraping currently is <a href="http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&amp;name=296-12602-1-ND" rel="nofollow noreferrer">http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&amp;name=296-12602-1-ND</a></p> <p>The code I'm currently using is pretty quick and dirty...</p> <pre><code> //This function retrieves data from the digikey private static List&lt;string&gt; ExtractProductInfo(HtmlDocument doc) { List&lt;HtmlNode&gt; m_unparsedProductInfoNodes = new List&lt;HtmlNode&gt;(); List&lt;string&gt; m_unparsedProductInfo = new List&lt;string&gt;(); //Base Node for part info string m_baseNode = @"//html[1]/body[1]/div[2]"; //Write part info to list m_unparsedProductInfoNodes.Add(doc.DocumentNode.SelectSingleNode(m_baseNode + @"/table[1]/tr[1]/td[1]/table[1]/tr[1]/td[1]")); //More lines of similar form will go here for more info //this retrieves digikey PN foreach(HtmlNode node in m_unparsedProductInfoNodes) { m_unparsedProductInfo.Add(node.InnerText); } return m_unparsedProductInfo; } </code></pre> <p>Although the path I'm using appears to be "correct" I keep getting NULL when I look at the list "m_unparsedProductInfoNodes"</p> <p>Any idea what's going on here? I'll also add that if I do a "SelectNodes" on the baseNode it only returns a div with the only significant child being "cs=####" which seems to vary with browser user agents. If I try to use this in anyway (putting /cs=0 in the path for the unidentifiable browser) it pitches a fit insisting that my expression doesn't evaluate to a node set, but leaving them still leaves the problem of all data past div[2] is returned as NULL.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload