osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Writes and Reads with high latency


hello all,

I have 1 DC of 3 nodes in which is running Cassandra 3.11.3 with consistency level ONE and Java 1.8.0_191.

Every day, there are many nodejs programs that send data to the cassandra's cluster via NodeJs cassandra-driver.
Every day I got like 600k requests. Each request makes the server to:
1_ READ some data in Cassandra (by an id, usually I get 3 records),
2_ DELETE one of those records
3_ WRITE the data into Cassandra.

So every day I make many deletes.

Every day I find errors like:
"All host(s) tried for query failed. First host tried, 10.8.0.10:9042: Host considered as DOWN. See innerErrors...."
"Server timeout during write query at consistency LOCAL_ONE (0 peer(s) acknowledged the write over 1 required)...."
"Server timeout during write query at consistency SERIAL (0 peer(s) acknowledged the write over 1 required)...."
"Server timeout during read query at consistency LOCAL_ONE (0 peer(s) acknowledged the read over 1 required)...."

nodetool tablehistograms tells me this:

Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)
50%             8.00            379.02           1955.67            379022                 8
75%            10.00            785.94         155469.30            654949                17
95%            12.00          17436.92         268650.95           1629722                35
98%            12.00          25109.16         322381.14           2346799                42
99%            12.00          30130.99         386857.37           3379391                50
Min             0.00              6.87             88.15               104                 0
Max            12.00          43388.63         386857.37          20924300               179

in the 99% I noted that write and read latency is pretty high, but I don't know how to improve that.
I can provide more statistics if needed.

Is there any improvement I can make to the Cassandra's configuration in order to not to lose any data?

Thanks 

Regards
Marco