Databases
StackOverflow2013
Postsdbo
PORails on Heroku: Generating image archives on the fly - improve performance
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORails on Heroku: Generating image archives on the fly - improve performance
    primarykey
    data
    text
    <p>We are building a Rails application on Heroku where users can upload images (which are stored on S3, US Standard region). Users can send friends a link where they can download the images as an archive (zip). This archive has to be generated on the fly - the content (i.e. which image) is different for each user accordant to some "magic rules".</p> <p>The problem: When a friend opens a download URL, it takes several seconds to build the archive. All images (about 4Mb in total) has to be downloaded from S3 to the Rails application, and then packed into an archive. Do you have ideas how to speed up this process, or can you help me evaluating the following options?</p> <ol> <li><strong>Asynchronous download</strong>: We could use an asynchrounous job queue for packaging (this won't make the process faster, but better scalable :) )</li> <li><strong>Cache on file system</strong>: We could cache the images on file system. But since web processes on Heroku doesn't necessarily share the same file systems, this have to be done for each new web process.</li> <li><strong>Parallelize downloads</strong>: The downloads are currently done sequentially, threads (ouch, better several job queues?) could be used for parallelization</li> <li><strong>Common caching component (memcached, redis)</strong>: Instead of caching on the file system, we could use memcached or redis for faster access (so we don't have to warm the caches for each web process).</li> </ol> <p>I made some brief benchmarks concerning the caching strategies, I try to download a 100Kb from different locations:</p> <ul> <li>File system: 0.0002s</li> <li>Memcached: 0.01s-0.09s</li> <li>Redis: 0.01s-0.09s </li> <li>S3: 0.1-0.25s</li> </ul> <p>So using a cache (file system, memcached or redis) would definitely improve the performance, the final solution would propably be to combine different approaches.</p> <p>Would do you think? Thank you very much for your help! </p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USlacco
    1. USlacco
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PORails on Heroku: Generating image archives on the fly - improve performance
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PORails on Heroku: Generating image archives on the fly - improve performance
      1. USShpigford
      1. VTFavorite
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload