Databases
StackOverflow2013
Postsdbo
PORuby Mechanize table scraping doesn't capture entire row
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORuby Mechanize table scraping doesn't capture entire row
    text
    copied!<p>I am trying to scrape a table website with mechanize. I want to scrape the second row.</p> <p>When I run :</p> <p><code>agent.page.search('table.ea').search('tr')[-2].search('td').map{ |n| n.text }</code></p> <p>I would expect it to scrape the whole row. But instead it only scrapes: ["2011-02-17", "0,00"]</p> <p>Why isn't it scraping all of the columns in the row, but just the first and the last column?</p> <p><strong>Xpath:</strong> /html/body/center/table/tbody/tr[2]/td[2]/table/tbody/tr[3]/td/table/tbody/tr[2]/td/table/tbody/tr[2]</p> <p><strong>CSS PATH:</strong> html body center table tbody tr td table tbody tr td table tbody tr td table.ea tbody tr td.total</p> <p>The page is similar to this:</p> <pre><code>&lt;table&gt;&lt;table&gt;&lt;table&gt; &lt;table width="100%" border="0" cellpadding="0" cellspacing="1" class="ea"&gt; &lt;tr&gt; &lt;th&gt;&lt;a href="#"&gt;Date&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;One&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;Two&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;Three&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;Four&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;Five&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;Six&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;Seven&lt;/a&gt;&lt;/th&gt; &lt;th&gt;&lt;a href="#"&gt;Eight&lt;/a&gt;&lt;/th&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;&lt;a href="#"&gt;2011-02-17&lt;/a&gt;&lt;/td&gt; &lt;td align="right"&gt;0&lt;/td&gt; &lt;td align="right"&gt;0&lt;/td&gt; &lt;td align="right"&gt;0,00&lt;/td&gt; &lt;td align="right"&gt;0&lt;/td&gt; &lt;td align="right"&gt;0&lt;/td&gt; &lt;td align="right"&gt;0&lt;/td&gt; &lt;td align="right"&gt;0&lt;/td&gt; &lt;td align="right"&gt;387&lt;/td&gt; &lt;td align="right"&gt;0,00&lt;/td&gt; &lt;!-- FOV --&gt; &lt;td align="right"&gt;0,00&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td class="total"&gt;Ialt&lt;/td&gt; &lt;td class="total" align="right"&gt;0&lt;/td&gt; &lt;td class="total" align="right"&gt;40&lt;/td&gt; &lt;td class="total" align="right"&gt;0,46&lt;/td&gt; &lt;td class="total" align="right"&gt;2&lt;/td&gt; &lt;td class="total" align="right"&gt;0&lt;/td&gt; &lt;td class="total" align="right"&gt;0&lt;/td&gt; &lt;td class="total" align="right"&gt;0&lt;/td&gt; &lt;td class="total" align="right"&gt;3.060&lt;/td&gt; &lt;td class="total" align="right"&gt;0,00&lt;/td&gt; &lt;td class="total" align="right"&gt;18,58&lt;/td&gt; &lt;/tr&gt; &lt;/table&gt; &lt;/table&gt;&lt;/table&gt;&lt;/table&gt; </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload