osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)


Thanks Mick. 


We're on datastax java driver version: 2.1.10.2 and we aren't using the client side timestamps. Anyway, we went ahead and verified that all client machines and cassandra machines are in sync with regards to time. 


We've also verified that no reads and writes are going to the remote data center. 


Here's a bit more information:

-Few rows in this column family can grow quite wide (> 100K columns)

-But we keep seeing this behavior most frequently with rows with just 1 or two columns . The typical behavior is: Machine A adds a new row and a column. 30-60 seconds later Machine B tries to read this row. It doesn't find the row. So the application retries within 500ms. This time it finds the row. 


Thanks.


From: Mick Semb Wever <mick@xxxxxxxxxxxxxxxxx>
Sent: Saturday, October 20, 2018 10:24:53 PM
To: user@xxxxxxxxxxxxxxxxxxxx
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
 
[ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ]

> Thanks James. Yeah, we're using the datastax java driver. But we're on version 2.1.10.2. And we are not using the client side timestamps.


Just to check Ninad. If you are using Cassandra-2.1 (native protocol
v3) and the java driver version 3.0 or above, then you would be using
client-side timestamps by default.
https://github.com/datastax/java-driver/tree/3.x/manual/query_timestamps

With client-side timestamps all client servers and all C* nodes must
be kept tightly in-sync, as Elliot said. Monitoring and alerting on
any clock skew on any of these machines is important.

Also worth checking that any local_quorum requests are not
accidentally go to the wrong datacenter.

regards,
Mick

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx

 

The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this e-mail and delete the message and any attachments from your computer.