Mick, sorry I think I missed your following questions:
We saw this issue a couple of times a few years ago. That's why we introduced this change. Although at that time, it was on cassandra 1.x version.
- Topology changes:
The only change we did was that we added nodes for 2 months. But that process finished 1 month ago. Since then we've run multiple repairs on the table. Other than that no other topology changes were done.
From: Mick Semb Wever <mick@xxxxxxxxxxxxxxxxx>
Sent: Wednesday, October 24, 2018 4:12:57 AM
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ]
> Here's a bit more information:
> -Few rows in this column family can grow quite wide (> 100K columns)
> -But we keep seeing this behavior most frequently with rows with just 1 or two columns . The typical behavior is: Machine A adds a new row and a column. 30-60 seconds later Machine B tries to read this row. It doesn't find the row. So the application retries within 500ms. This time it finds the row.
You wrote a lot of useful info in your original post, sorry I missed
it in my first reply.
Only thing there that stands out, apart from short reads that Jeff's
already pointed out, is the use of `speculative_retry='ALWAYS'`. Has
there topology changes in your cluster recently?
Next step would be to try and repeat it with tracing.
Mick Semb Wever
The Last Pickle
Apache Cassandra Consulting
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx
The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this e-mail and delete the message and any attachments from your computer.