Databases
StackOverflow2013
Postsdbo
POWhy is curl in Ruby slower than command-line curl?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhy is curl in Ruby slower than command-line curl?
    primarykey
    data
    text
    <p>I am trying to download more than 1m pages (URLs ending by a sequence ID). I have implemented kind of multi-purpose download manager with configurable number of download threads and one processing thread. The downloader downloads files in batches:</p> <pre><code>curl = Curl::Easy.new batch_urls.each { |url_info| curl.url = url_info[:url] curl.perform file = File.new(url_info[:file], "wb") file &lt;&lt; curl.body_str file.close # ... some other stuff } </code></pre> <p>I have tried to download 8000 pages sample. When using the code above, I get 1000 in 2 minutes. When I write all URLs into a file and do in shell:</p> <pre><code>cat list | xargs curl </code></pre> <p>I gen all 8000 pages in two minutes.</p> <p>Thing is, I need it to have it in ruby code, because there is other monitoring and processing code.</p> <p>I have tried:</p> <ul> <li>Curl::Multi - it is somehow faster, but misses 50-90% of files (does not download them and gives no reason/code)</li> <li>multiple threads with Curl::Easy - around the same speed as single threaded</li> </ul> <p>Why is reused Curl::Easy slower than subsequent command line curl calls and how can I make it faster? Or what I am doing wrong?</p> <p>I would prefer to fix my download manager code than to make downloading for this case in a different way.</p> <p>Before this, I was calling command-line wget which I provided with a file with list of URLs. Howerver, not all errors were handled, also it was not possible to specify output file for each URL separately when using URL list.</p> <p>Now it seems to me that the best way would be to use multiple threads with system call to 'curl' command. But why when I can use directly Curl in Ruby?</p> <p>Code for the download manager is here, if it might help: <a href="http://github.com/Stiivi/Datacamp-ETL/blob/master/lib/etl/download_manager.rb" rel="noreferrer" title="foo">Download Manager</a> (I have played with timeouts, from not-setting it to various values, it did not seem help)</p> <p>Any hints appreciated.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USStiivi
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POWhy is curl in Ruby slower than command-line curl?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POWhy is curl in Ruby slower than command-line curl?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POWhy is curl in Ruby slower than command-line curl?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI've tried to create simple curl system call instead of Curl::Easy: result = system("curl", "-s", "-o", path, url) And it seems to be much much faster. I am getting around 300kb/s instead of 60kb/s with Curl::Easy. It is weird - system call with huge system overhead and without connection reusing is much faster than library functions. CPU usage is much higher, obviously, however the speed is much better. Anyway, this is still not nice solution to my problem, even though working better.
      singulars
      1. POWhy is curl in Ruby slower than command-line curl?
      1. USStiivi
    2. COhave you tried using more recent versions of curb with Curl::Multi.download interface?
      singulars
      1. POWhy is curl in Ruby slower than command-line curl?
      1. UStodd
    3. CO`cat list | xargs curl` is passing many URLs to Curl on the command-line, instead of one at a time, so Curl is retrieving many at once. You can do that in Ruby easily enough, but you need to compare apples to apples and use HTTPClient or Typhoeus.
      singulars
      1. POWhy is curl in Ruby slower than command-line curl?
      1. USthe Tin Man
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload