Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>It all just depends on what you want to do with 404's.</p> <p>Lets assume that you just want to swallow them. Part of pguardiario's response is a good start: You can raise an error, and retry a few times...</p> <pre><code># lib/webpage_helper.rb require 'nokogiri' require 'open-uri' class WebpageHelper def self.get_doc(url) attempt_number = 0 begin attempt_number = attempt_number + 1 page_content = open(url).read # do more stuff rescue Exception =&gt; ex puts "Failed at #{Time.now}" puts "Error: #{ex}" puts "URL: " + url puts "Retrying... Attempt #: #{attempts.to_s}" sleep(10) retry if attempt_number &lt; 10 # Try ten times. end end end </code></pre> <p>If you followed this pattern, it would just fail silently. Nothing would happen, and it would move on after ten attempts. I would generally consider that a Bad Plan(tm). Instead of just failing out silently, I would go for something like this in the rescue clause:</p> <pre><code> rescue Exception =&gt; ex if attempt_number &lt; 10 # Try ten times. retry else raise "Unable to contact #{url} after ten tries." end end </code></pre> <p>and then throw something like this in MyHelperModule#do_the_process (you'd have to update your database to have an errors and error_message column):</p> <pre><code> my_models.each do |my_model| # ... cut ... begin some_var = SomeClass.new(my_model.id) rescue Exception =&gt; e my_model.update_attributes(errors: true, error_message: e.message) next end # ... cut ... end </code></pre> <p>That's probably the easiest and most graceful way to do it with what you currently have. That said, if you're handling that many request in one massive rake tasks, that's not very elegant. You can't restart it if something goes wrong, it's tying up a single process on your system for a long time, etc. If you end up with any memory leaks (or infinite loops!), you find yourself in a place where you can't just say 'move on'. You probably should be using some kind of queueing system like Resque or Sidekiq, or Delayed Job (though it sounds like you have more items that you'd end up queueing than Delayed Job would happily handle). I'd recommend digging in to those if you're looking for a more eloquent approach.</p>
    singulars
    1. This table or related slice is empty.
    1. POHow should my scraping "stack" handle 404 errors?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USChristopher WJ Rueber
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COChris, thank you. I was also thinking that using `raise` seemed like a good idea but i wasn't sure where it would go and where to `rescue` it. Do you think it would be prudent to use a custom/specific type of Exception?
      singulars
      1. PO
      1. USMario Zigliotto
    2. CO@SizzlePants: It's usually good practice to raise a custom child of `StandardError` rather than a generic, because frequently there's a gain to be had from raising one error while rescuing another. Also, it allows errors you didn't plan for to rise to the surface instead of going unnoticed. Furthermore, it's customary to create a higher-level umbrella class that can be used to catch all errors _specific to your application (or gem)_. Example: A gem "Foo" might define `class Foo::Error < StandardError` as an umbrella, and `class Foo::TimeoutError < Foo::Error` for the actual error to be raised.
      singulars
      1. PO
      1. USsinisterchipmunk
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload