We're on datastax java driver version: 22.214.171.124 and we aren't using the client side timestamps. Anyway, we went ahead and verified that all client machines and cassandra machines are in sync with regards to time.
We've also verified that no reads and writes are going to the remote data center.
Here's a bit more information:
-Few rows in this column family can grow quite wide (> 100K columns)
-But we keep seeing this behavior most frequently with rows with just 1 or two columns . The typical behavior is: Machine A adds a new row and a column. 30-60 seconds later Machine B tries to read this row. It doesn't find the row. So the application retries within 500ms. This time it finds the row.
From: Mick Semb Wever <mick@xxxxxxxxxxxxxxxxx>
Sent: Saturday, October 20, 2018 10:24:53 PM
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ]
> Thanks James. Yeah, we're using the datastax java driver. But we're on version 126.96.36.199. And we are not using the client side timestamps.
Just to check Ninad. If you are using Cassandra-2.1 (native protocol
v3) and the java driver version 3.0 or above, then you would be using
client-side timestamps by default.
With client-side timestamps all client servers and all C* nodes must
be kept tightly in-sync, as Elliot said. Monitoring and alerting on
any clock skew on any of these machines is important.
Also worth checking that any local_quorum requests are not
accidentally go to the wrong datacenter.
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx
The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this e-mail and delete the message and any attachments from your computer.