Databases
StackOverflow2013
Postsdbo
POMultithreaded file copy for Linux needed
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMultithreaded file copy for Linux needed
    primarykey
    data
    text
    <p>I have a "large" amount of data that needs to be copied every day. (6TB) </p> <p>It is 15 disks presented from a SAN over Fibre Channel and being copied to a local array consisting of 22 spindles. </p> <pre><code>sources are /mnt/disk1 /mnt/disk2 /mnt/disk3 destination is /mnt/Data/SystemBackup/ </code></pre> <p>Due to the nature of our SAN, single threaded file copy is not very fast; but it is very capable of 600+mb/sec if we ask it in the right way. :) I need a way to spawn multiple threads in a file copy. There are MANY ways to do this in windows... but I can't find any native methods available to Linux. </p> <p>Could something like Python or Perl be of assistance? Is there something I'm missing? What are your thoughts? </p> <p>Edit: (Please note, I am using a modified version of gnutils cp.) Read here for more info: <a href="http://www.usenix.org/event/lisa10/tech/slides/kolano.pdf" rel="nofollow">http://www.usenix.org/event/lisa10/tech/slides/kolano.pdf</a></p> <p>Edit2: The code</p> <pre><code>#!/bin/bash # Declare the foo OPTIONS="-r --double-buffer --threads=8" dstdir="/mnt/Data/PrUv2Backup/" mcp=/root/mcp # Cleanup old timestamp file rm -rf PrUv2CopyTimes.log # Declare array of source locations srcdirs=( PrUv2-home PrUv2-trax PrUv2-trax2 PrUv2-trax3 PrUv2-traxnl PrUv2-traxnl2 PrUv2-traxnl3 PrUv2-traxnl4 PrUv2-traxnv PrUv2-traxnv2 PrUv2-ulog PrUv2-zmops PrUv2-zmops2 PrUv2-zmops3 PrUv2-zmops4 ) for srcdir in "${srcdirs[@]}" do echo `date +"%r"` $srcdir start &gt;&gt; PrUv2CopyTimes.log $mcp $OPTIONS /mnt/$srcdir/ $dstdir echo `date +"%r"` $srcdir finish &gt;&gt; PrUv2CopyTimes.log done # email results cat PrUv2CopyTimes.log | mailx -r LouPrBoxen001 -s "Backup Complete" me@us.com </code></pre>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAaronJAnderson
    1. USAaronJAnderson
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POMultithreaded file copy for Linux needed
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhen you say "if we ask it in the right way", what exactly is "the right way"? I think the threading architecture critically depends on that.
      singulars
      1. POMultithreaded file copy for Linux needed
      1. USNPE
    2. COThis may sound silly, but in your copying script, couldn't you perform the copy operations in _parallel_? That ensures that your copy operations will effectively _max out_ your bandwidth. Of course you'll have to weigh the _number_ of operations you do in parallel vs performance et al but you get the idea.
      singulars
      1. POMultithreaded file copy for Linux needed
      1. USMarvin Pinto
    3. CO"The right way" is kind of an unknown to me. I know that it handles random IO in a wicked fast sort of way. It likes having lots of requests and lots to do... if you ask to copy a single large file, it just can't give you the IO.
      singulars
      1. POMultithreaded file copy for Linux needed
      1. USAaronJAnderson
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload