[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Inconsistent Quorum Read after Quorum Write

I’ve confirmed that the inconsistency issues disappeared after repair finished.

Anything changed with repair in 3.11.1? One difference I noticed is that the validation step during repair could turn down the node upon large tables, which never happen in 3.10. I had to throttle validation requests to let it pass. Also I switched back to -pr instead of incremental repair which is a resource killer and often hangs for the first node to be repaired.

To address the inconsistency issue, I could do Write All and Read One by giving up availability and stop running repair. Any comments on that?

I guess I could also try downgrade c* version but will data file be a problem?


On Jul 3, 2018, at 15:53, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:

Shouldn't happen. Any chance you could trace the queries, or have you been able to reproduce it? Also, what version of Cassandra?

On Wed., 4 Jul. 2018, 06:41 Visa, <liguilin2015@xxxxxxxxx> wrote:
Hi all,

We recently experienced an unexpected behavior with C* consistency.

For example, a table t consists of 4 columns - pk , a, b and c. We perform Quorum write and then Quorum read (RF=3 / LCS compaction).

The consistency seems to break while repairing is running(repair -pr).

Say, a record already exists in t like
pk=1, a=1, b=1, c=1

While repair is not running

Quorum Write:
update t set c = 2 where pk=1

Quorum Read:
select pk,a,b,c from t where pk=1 limit 1

Returns: (1, 1, 1, 2) as expected.

But if we do it while repair is running,

Quorum Write:
update t set c=3 where pk=1

Quorum Read, however, returns (1, null, null, 3) w/o values of a and b.

After repair is done, then the same Quorum Read returns the right values (1,1,1,3).

It does not happen to every row in t. The impacted rows are like 40 out of 300 millions. But still how the consistency gets broken here?

Thanks for your attention!

To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx