StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Here are some factors to consider along with the solution that I think handles it best:</p> <p><strong>Dev Time</strong> - <em>Datastore Admin</em> - To leverage Bulkloader, you'll need to write scripts, maintain backup servers, storage, etc.</p> <p><strong>Cost</strong> - <em>Datastore Admin</em> - YMMV but our backup of tens of millions of entities used <1% of the 1bil Task Queue quota. The cost for datastore read operations and storage will be specific to your application. But between the two options, the read operations should be the same and you're trading Outgoing Bandwidth ($0.12/GB) in Bulkloader for Blobstore storage ($0.0043/GB) with Datastore Admin. </p> <p><strong>Backup Duration</strong> - <em>Datastore Admin</em> - As you would expect, mapreduce shards writing data to Blobstore inside Google's network is much much faster than streaming the entity data out one at a time. A full backup of our data with Datastore Admin takes under 6 hours. With Bulkloader it takes over 3 days.</p> <p><strong>Backup Maintenance</strong> - <em>Bulkloader (for now)</em> - With Bulkloader and a server, you can create crons to regularly perform backups and backup maintenance. For example, we have a server in Rackspace that backs up our datastore every 3 days and keeps the last 2 backups. With Datastore Admin you have to manually perform the backup and delete stale backups, until an automated solution is published (<a href="http://code.google.com/p/googleappengine/issues/detail?id=7040" rel="noreferrer">Issue 7040</a>). Even still, for once a month backups, the cost of doing it manually with Datastore Admin is so low that I'd recommend it.</p> <p><strong>Data Flexibility</strong> - <em>Bulkloader</em> - With bulkloader you can export all your data into human-readable csv files allowing you to pivot it in Excel, create a test dataset for your local development environment, or even move your operation to another app hosting service (ex: AWS) should you require it.</p> <p><strong>Precision Restore</strong> - <em>Bulkloader</em> - Bulkloader can handle restoring select entities (where you know exactly which entities you deleted or overwrote) and bulk restores. Datastore Admin can only do bulk restore for all the entities of a given Kind.</p> <p><strong>Bulk Restore</strong> - <em>Datastore Admin</em> - Datastore Admin minimizes very expensive writes by only updating changed entities. Sharding also makes this process much, much faster than a simple Bulkloader upload (though you could shard the csv backup data across many clients yourself).</p> <p>Ultimately Bulkloader gives you more precise control while Datastore Admin simplifies and speeds up bulk backup/restore. Even though Datastore Admin is new and has a few issues (<a href="http://code.google.com/p/googleappengine/issues/detail?id=7076" rel="noreferrer">7076</a>), given your situation, I'd definitely recommend it.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload