Databases
StackOverflow2013
Postsdbo
POSevere degradation in Cassandra Write performance with continuous streaming data over time
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSevere degradation in Cassandra Write performance with continuous streaming data over time
    primarykey
    data
    text
    <p>I notice a severe degradation in Cassandra write performance with continuous writes over time. </p> <p>I am inserting time series data with time stamp (T) as the column name in a wide column that stores 24 hours worth of data in a single row. Streaming data is written from data generator (4 instances, each with 256 threads) inserting data into multiple rows in parallel. Additionally, data is also inserted into a column family that has indexes over DateType and UUIDType.</p> <p>CF1:</p> <pre><code> Col1 | Col2 | Col3(DateType) | Col(UUIDType4) | </code></pre> <p>RowKey1 </p> <p>RowKey2</p> <p>:</p> <p>:</p> <p>CF2 (Wide column family):</p> <p>RowKey1 (T1, V1) (T2, V3) (T4, V4) ......</p> <p>RowKey2 (T1, V1) (T3, V3) .....</p> <p>:</p> <p>:</p> <p>The no. of data points inserted/sec decreases over time until no further inserts are possible. The initial performance is of the order of 60000 ops/sec for ~6-8 hours and then it gradually tapers down to 0 ops/sec. Restarting the DataStax_Cassandra_Community_Server on all nodes helps restore the original throughput, but the behaviour is observed again after a few hours. </p> <p>OS: Windows Server 2008 No.of nodes: 5 Cassandra version: DataStax Community 1.2.3 RAM: 8GB HeapSize: 3GB Garbage collector: default settings [ParNewGC]</p> <p>I also notice a phenomenal increase in the no. of Pending write requests as reported by the OpsCenter (~of magnitude 200,000) when the performance begins to degrade.</p> <p>I fail to understand what is preventing the write operations to be completed and why do they pile up over time? I do not see anything suspicious in the Cassandra logs. </p> <p>Has the OS settings got anything to do with this? Any suggestions to probe this issue further?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USvinay sudhakar
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSevere degradation in Cassandra Write performance with continuous streaming data over time
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI would upgrade to the latest 1.2.X release, 1.2.12, and then try your test again. There have been a lot of bug fixes since 1.2.3
      singulars
      1. POSevere degradation in Cassandra Write performance with continuous streaming data over time
      1. USZanson
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload