StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POCopying a 1TB sparse file
text
Body
copied!<p>I got a sparse file of 1TB which stores actually 32MB data on Linux.</p> <p>Is it possible to "efficiently" make a package to store the sparse file? The package should be unpacked to be a 1TB sparse file on another computer. Ideally, the "package" should be around 32MB.</p> <p>Note: On possible solution is to use 'tar': <a href="https://wiki.archlinux.org/index.php/Sparse_file#Archiving_with_.60tar.27" rel="nofollow noreferrer">https://wiki.archlinux.org/index.php/Sparse_file#Archiving_with_.60tar.27</a></p> <p>However, for a 1TB sparse file, although the tar ball may be small, archiving the sparse file will take too long a time.</p> <p><strong>Edit 1</strong></p> <p>I tested the tar and gzip and the results are as follows (Note that this sparse file contains data of 0 byte).</p> <pre><code>$ du -hs sparse-1 0 sparse-1 $ ls -lha sparse-1 -rw-rw-r-- 1 user1 user1 1.0T 2012-11-03 11:17 sparse-1 $ time tar cSf sparse-1.tar sparse-1 real 96m19.847s user 22m3.314s sys 52m32.272s $ time gzip sparse-1 real 200m18.714s user 164m33.835s sys 10m39.971s $ ls -lha sparse-1* -rw-rw-r-- 1 user1 user1 1018M 2012-11-03 11:17 sparse-1.gz -rw-rw-r-- 1 user1 user1 10K 2012-11-06 23:13 sparse-1.tar </code></pre> <p>The 1TB file sparse-1 which contains 0 byte data can be archived by 'tar' to a 10KB tar ball or compressed by gzip to a ~1GB file. gzip takes around 2 times of the time than the time tar uses.</p> <p>From the comparison, 'tar' seems better than gzip.</p> <p>However, 96 minutes are too long for a sparse file that contains data of 0 byte.</p> <p><strong>Edit 2</strong></p> <p><code>rsync</code> seems finish copying the file in more time than <code>tar</code> but less than <code>gzip</code>:</p> <pre><code>$ time rsync --sparse sparse-1 sparse-1-copy real 124m46.321s user 107m15.084s sys 83m8.323s $ du -hs sparse-1-copy 4.0K sparse-1-copy </code></pre> <p>Hence, <code>tar</code> + <code>cp</code> or <code>scp</code> should be faster than directly <code>rsync</code> for this extremely sparse file.</p> <p><strong>Edit 3</strong></p> <p>Thanks to @mvp for pointing out the SEEK_HOLE functionality in newer kernel. (I previously work on a 2.6.32 Linux kernel).</p> <p>Note: bsdtar version >=3.0.4 is required (check here: <a href="http://ask.fclose.com/4/how-to-efficiently-archive-a-very-large-sparse-file?show=299#c299" rel="nofollow noreferrer">http://ask.fclose.com/4/how-to-efficiently-archive-a-very-large-sparse-file?show=299#c299</a> ).</p> <p>On a newer kernel and Fedora release (17), <code>tar</code> and <code>cp</code> handles the sparse file <strong>very</strong> efficiently.</p> <pre><code>[zma@office tmp]$ ls -lh pmem-1 -rw-rw-r-- 1 zma zma 1.0T Nov 7 20:14 pmem-1 [zma@office tmp]$ time tar cSf pmem-1.tar pmem-1 real 0m0.003s user 0m0.003s sys 0m0.000s [zma@office tmp]$ time cp pmem-1 pmem-1-copy real 0m0.020s user 0m0.000s sys 0m0.003s [zma@office tmp]$ ls -lh pmem* -rw-rw-r-- 1 zma zma 1.0T Nov 7 20:14 pmem-1 -rw-rw-r-- 1 zma zma 1.0T Nov 7 20:15 pmem-1-copy -rw-rw-r-- 1 zma zma 10K Nov 7 20:15 pmem-1.tar [zma@office tmp]$ mkdir t [zma@office tmp]$ cd t [zma@office t]$ time tar xSf ../pmem-1.tar real 0m0.003s user 0m0.000s sys 0m0.002s [zma@office t]$ ls -lha total 8.0K drwxrwxr-x 2 zma zma 4.0K Nov 7 20:16 . drwxrwxrwt. 35 root root 4.0K Nov 7 20:16 .. -rw-rw-r-- 1 zma zma 1.0T Nov 7 20:14 pmem-1 </code></pre> <p>I am using a 3.6.5 kernel:</p> <pre><code>[zma@office t]$ uname -a Linux office.zhiqiangma.com 3.6.5-1.fc17.x86_64 #1 SMP Wed Oct 31 19:37:18 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload