Databases
StackOverflow2013
Postsdbo
POCassandra Compaction takes all the resources and leads to node failure
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POCassandra Compaction takes all the resources and leads to node failure
    text
    copied!<p>I met very strange problem during testing cassandra. I have a very simple column family that stores video data (keys point to time period and there is only one column containing ~2MB video for this period).</p> <p><strong>Use Case</strong></p> <p>I start to load data using Hector API (round-robin) to 6 empty nodes (8GB RAM for Cassandra)- load is run in 4 threads adding 4 rows in second for each thread.</p> <p>After a while (running load for hour or so) near 100-200 GB are added to the node (depending on the replication factor) and then one or several nodes become unreachable. (no pinging just reboot helps)</p> <p><strong>Why Compaction</strong></p> <p>I do use tiered-level compaction and monitoring the system(Debian) i can see that it actually not writes but compaction that takes almost all resources (disk, memory) and cause server to refuse writes and than fail.</p> <p>After like 30-40 minutes of test compaction tasks just cannot be handled and get queued. Interesting thing is that there are no deletes and updates - so compaction just reads/writes data again and again without bringing actual value to me (like it can be compacted once in the evening).</p> <p>When i slow down the pace - i.e running 2 threads with 1 second delay things go better but whether it still be working when i have 20TB not 100 GB on a node.</p> <p>Is Cassandra optimized for such type of workload? How the resources are normally distributed between compaction and reads/writes?</p> <p><strong>Update</strong> Update of network driver solved problem with unreachable cluster </p> <p>Thanks, Sergey.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload