Databases
StackOverflow2013
Postsdbo
PONeed to change the storage "directory" of files in an S3 Bucket (Carrierwave / Fog)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONeed to change the storage "directory" of files in an S3 Bucket (Carrierwave / Fog)
    primarykey
    data
    text
    <p>I am using Carrierwave with 3 separate models to upload photos to S3. I kept the default settings for the uploader, which was to store photos in a root S3 bucket. I then decided to store them in sub-directories according to model name like /avatars, items/, etc. based on the model they were uploaded from...</p> <p>Then, I noticed that files of the same name were being overwritten and when I deleted a model record, the photo wasn't being deleted. </p> <p>I've since changed the store_dir from an uploader-specific setup like this:</p> <pre><code> def store_dir "items" end </code></pre> <p>to a generic one which stores photo under the model ID (I use mongo FYI):</p> <pre><code> def store_dir "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}" end </code></pre> <p>Here comes the problem. I am trying to move all the photos already into S3 into the proper "directory" within S3. From what I've ready, S3 doesn't have directories per se. I'm having trouble with the rake task. Since i changed the store_dir, Carrierwave is looking for all the photos previously uploaded in the wrong directory.</p> <pre><code>namespace :pics do desc "Fix directory location of pictures on s3" task :item_update =&gt; :environment do connection = Fog::Storage.new({ :provider =&gt; 'AWS', :aws_access_key_id =&gt; 'XXXX', :aws_secret_access_key =&gt; 'XXX' }) directory = connection.directories.get("myapp-uploads-dev") Recipe.all.each do |l| if l.images.count &gt; 0 l.items.each do |i| if i.picture.path.to_s != "" new_full_path = i.picture.path.to_s filename = new_full_path.split('/')[-1].split('?')[0] thumb_filename = "thumb_#{filename}" original_file_path = "items/#{filename}" puts "attempting to retrieve: #{original_file_path}" original_thumb_file_path = "items/#{thumb_filename}" photo = directory.files.get(original_file_path) rescue nil if photo puts "we found: #{original_file_path}" photo.expires = 2.years.from_now.httpdate photo.key = new_full_path photo.save thumb_photo = directory.files.get(original_thumb_file_path) rescue nil if thumb_photo puts "we found: #{original_thumb_file_path}" thumb_photo.expires = 2.years.from_now.httpdate thumb_photo.key = "/uploads/item/picture/#{i.id}/#{thumb_filename}" thumb_photo.save end end end end end end end end </code></pre> <p>So I'm looping through all the Recipes, looking for items with photos, determining the old Carrierwave path, trying to update it with the new one based on the store_dir change. I thought if I simply updated the photo.key with the new path, it would work, but it's not.</p> <p>What am I doing wrong? Is there a better way to accomplish the ask here?</p> <p><strong>Here's what I did to get this working...</strong></p> <pre><code>namespace :pics do desc "Fix directory location of pictures" task :item_update =&gt; :environment do connection = Fog::Storage.new({ :provider =&gt; 'AWS', :aws_access_key_id =&gt; 'XXX', :aws_secret_access_key =&gt; 'XXX' }) bucket = "myapp-uploads-dev" puts "Using bucket: #{bucket}" Recipe.all.each do |l| if l.images.count &gt; 0 l.items.each do |i| if i.picture.path.to_s != "" new_full_path = i.picture.path.to_s filename = new_full_path.split('/')[-1].split('?')[0] thumb_filename = "thumb_#{filename}" original_file_path = "items/#{filename}" original_thumb_file_path = "items/#{thumb_filename}" puts "attempting to retrieve: #{original_file_path}" # copy original item begin connection.copy_object(bucket, original_file_path, bucket, new_full_path, 'x-amz-acl' =&gt; 'public-read') puts "we just copied: #{original_file_path}" rescue puts "couldn't find: #{original_file_path}" end # copy thumb begin connection.copy_object(bucket, original_thumb_file_path, bucket, "uploads/item/picture/#{i.id}/#{thumb_filename}", 'x-amz-acl' =&gt; 'public-read') puts "we just copied: #{original_thumb_file_path}" rescue puts "couldn't find thumb: #{original_thumb_file_path}" end end end end end end end </code></pre> <p>Perhaps not the prettiest thing in the world, but it worked.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USaressidi
    1. USaressidi
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PONeed to change the storage "directory" of files in an S3 Bucket (Carrierwave / Fog)
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI would kind of expect that to work as well. Are there errors or the files just don't exist where you expect? If there are small number this may work fine, but especially with a larger number you may want to use copy_object as jeremy mentions below (as it does everything much more quickly and you don't have to download anything).
      singulars
      1. PONeed to change the storage "directory" of files in an S3 Bucket (Carrierwave / Fog)
      1. USgeemus
    2. COThanks for this, it's been helpful for me going through the same thing. One thing you might want to know in the future is you can get the filename directly without having to parse the path: i.picture.file.filename would do it in your case.
      singulars
      1. PONeed to change the storage "directory" of files in an S3 Bucket (Carrierwave / Fog)
      1. USNick Gottlieb
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload