Databases
StackOverflow2013
Postsdbo
POBest solution for a caching system that supports sharding, replication and has low latency
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POBest solution for a caching system that supports sharding, replication and has low latency
    text
    copied!<p>We're in the process of deploying a highly dynamic website. About 20,000 items are processed and updated every minute at peak capacity. Each item can range from a size of 1kb to 500kb. These items needs to be retrieved, processed and updated in cache every minute.</p> <p>We are expecting a traffic of upto 1000 users in the first two-three months. As each user lands on the website, they can be requesting some popular content, but others may request unpopular content. All content is a higher level processed form of what sits in the persistent store. Hence it is absolutely necessary to have all the processed items sitting in a low-latency store for superb user-experience, be it popular or unpopular.</p> <p>We've tried Memcache, Redis and Couchbase separately. </p> <p>Memcache is super fast but we ran into issues where certain slabs ran out of memory and active items started getting evicted.</p> <p>Redis, relatively slower than Memcache, is great if you want persistence in the items. </p> <p>However soon we realized we wanted sharding and replication. </p> <p>Couchbase offered that out of the box.. The Moxi-client that interfaces with the Couchbase server has its own problems of not being able to handle heavy concurrent processes. It will start missing sets and gets every now and then. Moved over to the Python SDK that interfaces with it. It performed poorly in the event when one of the nodes in the cluster went down, it wasn't able to discover the new topology at all. Ended up losing some data in cache and inactivity on the site for several precious hours.</p> <p>At a point where we realize that there is no perfect product out there that will suite our needs. You have to be aware of all the technologies and your own needs. You have to foresee how your data will evolve and be prepared accordingly. The best solution is probably a hybrid of technologies. However putting this out in the hope that maybe there is something other there. We're approaching the end of 2012. How hard can it be for an out of the box solution backed up with powerful hardware to deliver what we need. </p> <p>Any thoughts and links to insightful articles would be greatly appreciated. Thanks!</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload