Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Dear stack overflow community,</p> <p>as promised here are the results of the experiments done based on your many suggestions. A special thanks to @user894763 how put me on the "right path".</p> <blockquote> <p>tl;dr use <em>pnm files inside an uncompressed tar</em> (yes I said pnm !).</p> </blockquote> <p>I have done experiments on two high end machines, one enabled with SSD disks, and the other one using a networked file system. Both have high end CPUs, but show "two side of the spectrum" on disk access. Surprisingly, the conclusions are the same for both machines. I report only one set of results (for the later case). The ratios amongst file formats are almost identical in both experiments.</p> <p>From these experiments I have learned two important things:</p> <ul> <li>When regarding files from disk, the operative system disk cache is king (i.e. the operative systems tries as much as possible to keep file operations in RAM instead of the physical device, and it does a really good job at this).</li> <li>Contrary to my initial guess, reading images from disk is a CPU bounded operation, not an I/O bounded.</li> </ul> <h3>Experiment protocol</h3> <p>I am reading a set of ~1200 images in a fix sequence, no computation is done on the images, I am simply measuring the time to load the pixels in memory. The tar files sizes are ~600 MB in pnm format, ~300 MB in png format, and ~200 MB in webp format.</p> <p>"Fresh read" means first read done on the machine.<br> "Cached read" means the second read done on the same machine (and any subsequent one).</p> <p>All numbers are roughly +- 10 Hz.</p> <pre><code>webp fresh read: 30 Hz webp cached read: 80 Hz webp + tar fresh read: 100 Hz webp + tar cached read: 100 Hz png fresh read: 50 Hz png cached read: 165 Hz png + tar fresh read: 200 Hz png + tar cached read: 200 Hz pnm fresh read: 50 Hz pnm cached read: 600 Hz pnm + tar fresh read: 200 Hz pnm + tar cached read: 2300 Hz </code></pre> <h3>Notes</h3> <p>I was told that maybe there is way to change the webp compression parameters to make the decompression faster. I suspect that it would still not match the pnm performance.</p> <p>Please note that I used custom code to read the images in the tar file, the file is read from disk "image by image".</p> <p>I do not know why reading the webp images "fresh" was slower than the png ones, I can only speculate that the networked disk system had some "internal" cache that somewhat changed the behaviour. However this does not affect the lessons.</p> <h3>Lessons</h3> <ol> <li><p>If you will read a file (or a set of files) multiple times, the operative system disk cache will make all future reads essentially "as fast as reading from RAM".</p></li> <li><p>Even when reading from disk the time to decompress images is non-negligible.</p></li> <li><p>Putting all the files into single uncompressed (tar) file, makes things significantly faster because the operative system will assume that the whole file will be read, pre-loading future images even before we access them. This seem not to happen when simply reading inside a folder.</p></li> <li><p>With proper care, a factor 4x ~ x10 in speed-up can be obtained when reading a sequence of images from disk (specially if read repeatedly).</p></li> </ol>
    singulars
    1. This table or related slice is empty.
    1. POWhat is the fastest way to read a sequence of images?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USrodrigob
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POWhat is the fastest way to read a sequence of images?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COFollowing https://groups.google.com/a/webmproject.org/forum/?fromgroups=#!topic/webp-discuss/FPOfZs2cCS4 tried "cwebp -preset photo -q 100" instead of "cwebp -preset photo -lossless". The new tar file now is ~100 MB, reading webp files fresh is 90 Hz, 105 Hz cached. From tar file it is 115 Hz (both fresh and cached). Better, but not better than pnm + tar (plus we may have introduced minor compression artefacts).
      singulars
      1. PO
      1. USrodrigob
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload