OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Failed request after changing compression strategy from LZ4 to Deflate and using upgradesstable


Hi Hitesh,
This list is for conversations regarding Cassandra development. Can
you subscribe and post this to users@xxxxxxxxxxxxxxxxxxxx instead? You
will get a much wider audience when you do.



On Tue, Apr 10, 2018 at 10:45 PM, hitesh dua <hiteshdua1@xxxxxxxxx> wrote:
>  Hi,
> My Compression strategy in Production was *LZ4 Compression. *But I modified
> it to *Deflate *using alter command
>
> For compression change, we have to use *nodetool Upgradesstables *to
> forcefully upgrade the compression strategy on all sstables
>
> But once upgradesstabloes command completed on all the 5 nodes in the
> cluster, My requests started to fail, both read and write
>
> Replication Factor - 3
> Read Consistency - 1
> Write Consistency - 1
> FYI - I am also using lightweight transaction which uses PAXOS
> Cassandra Version 3.10
>
> I am now facing Following Errors in my debug.log file and some of my
> requests have started to fail :
>
> Debug.log
>
> ERROR [ReadRepairStage:82952] 2018-04-09 19:05:20,669
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[ReadRepairStage:82952,5,main]
>>
>> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
>>> - received only 0 responses.
>>
>> at org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat
>>> ors$2.close(UnfilteredPartitionIterators.java:182)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr
>>> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> ~[na:1.8.0_144]
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> ~[na:1.8.0_144]
>>
>> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$
>>> threadLocalDeallocator$0(NamedThreadFactory.java:79)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
>>
>> DEBUG [ReadRepairStage:82953] 2018-04-09 19:05:22,932
>>> ReadCallback.java:242 - Digest mismatch:
>>
>> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
>>> DecoratedKey(-2666936192316364820, 5756f5b8e7b341afa22cef22c5d33260)
>>> (d29a0e2a05f81315f0945dee5a210060 vs d41d8cd98f00b204e9800998ecf8427e)
>>
>> at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> [na:1.8.0_144]
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> [na:1.8.0_144]
>>
>> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$
>>> threadLocalDeallocator$0(NamedThreadFactory.java:79)
>>> [apache-cassandra-3.10.jar:3.10]
>>
>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
>>
>> INFO  [HintsDispatcher:767] 2018-04-09 19:05:24,874
>>> HintsDispatchExecutor.java:283 - Finished hinted handoff of file
>>> 68c7c130-6cf8-4864-bde8-1819f238045c-1523315072851-1.hints to endpoint
>>> 68c7c130-6cf8-4864-bde8-1819f238045c, partially
>>
>> DEBUG [ReadRepairStage:82950] 2018-04-09 19:05:24,932
>>> DataResolver.java:169 - Timeout while read-repairing after receiving all 1
>>> data and digest responses
>>
>> ERROR [ReadRepairStage:82950] 2018-04-09 19:05:24,933
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[ReadRepairStage:82950,5,main]
>>
>> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
>>> - received only 0 responses.
>>
>> at org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat
>>> ors$2.close(UnfilteredPartitionIterators.java:182)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr
>>> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10]
>>
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> ~[na:1.8.0_144]
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> ~[na:1.8.0_144]
>>
>> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$
>>> threadLocalDeallocator$0(NamedThreadFactory.java:79)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>
>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
>>
>>
> nodetool info shows
>
> Gossip active          : true
>>
>> Thrift active          : false
>>
>> Native Transport active: true
>>
>> Load                   : 280.43 GiB
>>
>> Generation No          : 1514537104
>>
>> Uptime (seconds)       : 8810363
>>
>> Heap Memory (MB)       : 1252.06 / 3970.00
>>
>> Off Heap Memory (MB)   : 573.33
>>
>> Data Center            : dc1
>>
>> Rack                   : rack1
>>
>> *Exceptions             : 18987*
>>
>> Key Cache              : entries 351612, size 99.86 MiB, capacity 100 MiB,
>>> 11144584 hits, 21126425 requests, 0.528 recent hit rate, 14400 save period
>>> in seconds
>>
>>
> Hope anyone can help me with this.
>
> Thanks,
> Hitesh dua
> hiteshdua1@xxxxxxxxx

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx