Databases
StackOverflow2013
Postsdbo
POSkipping slow websites when looping through an array of URLs using watir-webdriver
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSkipping slow websites when looping through an array of URLs using watir-webdriver
    text
    copied!<p>I'm trying to loop through an array of websites in Chrome using watir-webdriver, but I always encounter an error on certain websites. Recently, I have had this problem with <a href="http://adage.com" rel="nofollow">http://adage.com</a>. The loop will execute perfectly until it reaches <a href="http://adage.com" rel="nofollow">http://adage.com</a> and then it will hang until the following error is displayed:</p> <pre><code>/Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill': Timeout::Error (Timeout::Error) from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:132:in `readline' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:2551:in `read_new' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1316:in `catch' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1293:in `request' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1286:in `block in request' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:745:in `start' from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1284:in `request' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/http/default.rb:82:in `response_for' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/http/default.rb:38:in `request' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/http/common.rb:40:in `call' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/bridge.rb:598:in `raw_execute' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/bridge.rb:576:in `execute' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/bridge.rb:536:in `getActiveElement' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/common/target_locator.rb:60:in `active_element' from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/watir-webdriver-0.6.1/lib/watir-webdriver/browser.rb:136:in `send_keys' from /Users/default/Dropbox/beta_scripts/loop_test.rb:16:in `rescue in &lt;main&gt;' from /Users/default/Dropbox/beta_scripts/loop_test.rb:11:in `&lt;main&gt;' </code></pre> <p>I have no idea how to avoid this. I have tried setting timeouts and even sending the <code>ESC</code> key during rescue to stop Chrome from loading the page, but have not had any success. Ultimately, I want to be able to <em>reliably</em> load an array of 500+ websites in succession, but this seems impossible given the likelihood that one of the websites will hang. <strong>Is there any way to stop a slow page from loading and move on to the next element in the array?</strong></p> <p>Below is a shortened version of my code that isolates the problem:</p> <pre><code>#!/usr/bin/env ruby require 'watir-webdriver' b = Watir::Browser.new :chrome sites = ["twitter.com", "cars.com", "autotrader.com", "rolex.com", "newyorker.com", "adage.com", "theatlantic.com", "pcmag.com"] sites.each do |uri| begin Timeout::timeout(10) do b.goto uri end rescue Timeout::Error =&gt; e_time sleep 5 b.send_keys :escape p "#{uri} is taking forever to load (#{e_time})" rescue Exception =&gt; e_exception p e_exception end end b.close </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload