sounds like the foreground read repair can cause issues to twcs (mix old and new data in same sstable), is there a way to disable the foreground read repair? is that indeed the case that it's problematic?On Mon, Dec 17, 2018 at 9:21 AM Gil Ganz <gilganz@xxxxxxxxx> wrote:hey jeff, attaching more information.so this the situation before - 3 nodes in the cluster (3.11.3 in this case but i saw same thing in 2.1 and 3.0), there is a script writing one row every minute and another script doing nodetool flush every 10 minute.
window is defined as two hours, so after a few days this is how the directory listing looks :drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups-rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db-rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db-rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db-rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db-rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32-rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 mc-171-big-CompressionInfo.db-rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25 mc-171-big-Statistics.db-rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt-rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db-rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db-rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db-rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db-rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32-rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 mc-172-big-CompressionInfo.db-rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25 mc-172-big-Statistics.db-rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt-rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db-rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db-rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db-rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db-rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32-rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 mc-173-big-CompressionInfo.db-rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt-rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25 mc-173-big-Statistics.db..-rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db-rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db-rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db-rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db-rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32-rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10 mc-873-big-CompressionInfo.db-rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10 mc-873-big-Statistics.db-rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt...-rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db-rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db-rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db-rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50 mc-1150-big-Digest.crc32-rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db-rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50 mc-1150-big-CompressionInfo.db-rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt-rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50 mc-1150-big-Statistics.dbcheck out sstable 873 is from dec 15. at this point i remove node no 3.and the log shows streamed data is creating sstables 1150-1200. it also shows one of these sstables being compacted with sstable 873, resulting in sstable number 1228, an operation which from what i understand should not happen.DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199 StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136] Received 58 sstables from /10.106.12.34 ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564 TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in current bucket, compacting what's here: [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564 CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136) [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0, /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0, ]DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578 CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136) 2 sstables to [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,] to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput = ~184/s. 71 total partitions merged to 71. Partition merge counts were {1:71, }see sstablemetadata output for sstable 873 and 1228. as you can see the 1228 min timstamp is old.873:WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshotsSSTable: /root/gil_sstables_before/mc-873-bigPartitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000Minimum timestamp: 1544896864561707Maximum timestamp: 1544903882074190SSTable min local deletion time: 2147483647SSTable max local deletion time: 2147483647Compressor: org.apache.cassandra.io.compress.LZ4CompressorCompression ratio: 0.6954296160877513TTL min: 0TTL max: 0First token: -7160136740246525330 (key=42)Last token: 8209703677368372043 (key=34)minClustringValues: [2018-12-15 18:01Z]maxClustringValues: [2018-12-15 19:58Z]Estimated droppable tombstones: 0.0SSTable Level: 0Repaired at: 0Replay positions covered: {CommitLogPosition(segmentId=1544706489001, position=19800215)=CommitLogPosition(segmentId=1544706489001, position=20586499)}totalColumnsSet: 62totalRows: 62Estimated tombstone drop times:Count Row Size Cell Count1 0 242 0 73 0 54 0 15 0 16 0 07 0 08 0 010 0 012 0 014 0 017 0 020 0 024 0 029 0 035 0 042 0 050 7 060 17 072 0 086 5 0103 2 0124 5 0149 1 0179 0 0215 1 0258 0 0310 0 0372 0 0446 0 0535 0 0642 0 0770 0 0924 0 01109 0 01331 0 01597 0 01916 0 02299 0 02759 0 03311 0 03973 0 04768 0 05722 0 06866 0 08239 0 09887 0 011864 0 014237 0 017084 0 020501 0 024601 0 029521 0 035425 0 042510 0 051012 0 061214 0 073457 0 088148 0 0105778 0 0126934 0 0152321 0 0182785 0 0219342 0 0263210 0 0315852 0 0379022 0 0454826 0 0545791 0 0654949 0 0785939 0 0943127 0 01131752 0 01358102 0 01629722 0 01955666 0 02346799 0 02816159 0 03379391 0 04055269 0 04866323 0 05839588 0 07007506 0 08409007 0 010090808 0 012108970 0 014530764 0 017436917 0 020924300 0 025109160 0 030130992 0 036157190 0 043388628 0 052066354 0 062479625 0 074975550 0 089970660 0 0107964792 0 0129557750 0 0155469300 0 0186563160 0 0223875792 0 0268650950 0 0322381140 0 0386857368 0 0464228842 0 0557074610 0 0668489532 0 0802187438 0 0962624926 0 01155149911 0 01386179893 0 01663415872 0 01996099046 0 02395318855 0 02874382626 03449259151 04139110981 04966933177 05960319812 07152383774 08582860529 010299432635 012359319162 014831182994 017797419593 021356903512 025628284214 030753941057 036904729268 044285675122 053142810146 063771372175 076525646610 091830775932 0110196931118 0132236317342 0158683580810 0190420296972 0228504356366 0274205227639 0329046273167 0394855527800 0473826633360 0568591960032 0682310352038 0818772422446 0982526906935 01179032288322 01414838745986 0Estimated cardinality: 38EncodingStats minTTL: 0EncodingStats minLocalDeletionTime: 1442880000EncodingStats minTimestamp: 1544896864561707KeyType: org.apache.cassandra.db.marshal.Int32TypeClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]StaticColumns: {}RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}You have new mail in /var/spool/mail/root1228 :WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshotsSSTable: /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-bigPartitioner: org.apache.cassandra.dht.Murmur3PartitionerBloom Filter FP chance: 0.010000Minimum timestamp: 1544896805142916Maximum timestamp: 1544903882074190SSTable min local deletion time: 2147483647SSTable max local deletion time: 2147483647Compressor: org.apache.cassandra.io.compress.LZ4CompressorCompression ratio: 0.6942828485456369TTL min: 0TTL max: 0First token: -9108684050423740263 (key=53)Last token: 8669466869824630438 (key=77)minClustringValues: [2018-12-15 18:00Z]maxClustringValues: [2018-12-15 19:58Z]Estimated droppable tombstones: 0.0SSTable Level: 0Repaired at: 0Replay positions covered: {CommitLogPosition(segmentId=1544706489001, position=19800215)=CommitLogPosition(segmentId=1544706489001, position=20586499)}totalColumnsSet: 113totalRows: 113Estimated tombstone drop times:Count Row Size Cell Count1 0 412 0 213 0 74 0 15 0 16 0 07 0 08 0 010 0 012 0 014 0 017 0 020 0 024 0 029 0 035 0 042 0 050 16 060 25 072 0 086 17 0103 4 0124 7 0149 1 0179 0 0215 1 0258 0 0310 0 0372 0 0446 0 0535 0 0642 0 0770 0 0924 0 01109 0 01331 0 01597 0 01916 0 02299 0 02759 0 03311 0 03973 0 04768 0 05722 0 06866 0 08239 0 09887 0 011864 0 014237 0 017084 0 020501 0 024601 0 029521 0 035425 0 042510 0 051012 0 061214 0 073457 0 088148 0 0105778 0 0126934 0 0152321 0 0182785 0 0219342 0 0263210 0 0315852 0 0379022 0 0454826 0 0545791 0 0654949 0 0785939 0 0943127 0 01131752 0 01358102 0 01629722 0 01955666 0 02346799 0 02816159 0 03379391 0 04055269 0 04866323 0 05839588 0 07007506 0 08409007 0 010090808 0 012108970 0 014530764 0 017436917 0 020924300 0 025109160 0 030130992 0 036157190 0 043388628 0 052066354 0 062479625 0 074975550 0 089970660 0 0107964792 0 0129557750 0 0155469300 0 0186563160 0 0223875792 0 0268650950 0 0322381140 0 0386857368 0 0464228842 0 0557074610 0 0668489532 0 0802187438 0 0962624926 0 01155149911 0 01386179893 0 01663415872 0 01996099046 0 02395318855 0 02874382626 03449259151 04139110981 04966933177 05960319812 07152383774 08582860529 010299432635 012359319162 014831182994 017797419593 021356903512 025628284214 030753941057 036904729268 044285675122 053142810146 063771372175 076525646610 091830775932 0110196931118 0132236317342 0158683580810 0190420296972 0228504356366 0274205227639 0329046273167 0394855527800 0473826633360 0568591960032 0682310352038 0818772422446 0982526906935 01179032288322 01414838745986 0Estimated cardinality: 71EncodingStats minTTL: 0EncodingStats minLocalDeletionTime: 1442880000EncodingStats minTimestamp: 1544896805142916KeyType: org.apache.cassandra.db.marshal.Int32TypeClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]StaticColumns: {}RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}thanksgil---------- Forwarded message ---------
From: Jeff Jirsa <jjirsa@xxxxxxxxx>
Date: Fri, Dec 14, 2018 at 8:56 AM
Subject: Re: TWCS sstables gets merged following node removal
To: <user@xxxxxxxxxxxxxxxxxxxx>Remove node will stream data from all windows to remote nodes , so some compaction is expectedWould need to see the sstablemetadata to understand what’s happening there.--Jeff JirsaHi all ,
My colleague opened Jira ticket for the issue but we are struggling with this issue for a while and we have space issues :
https://issues.apache.org/jira/browse/CASSANDRA-14929
After removing a node from the cluster, a table that is defined as TWCS, has sstables from different time windows merged together, making old and new data sit in the same sstable(the full example is in the ticket).
Any idea about this?workaround?Known issues?
Thanks!
Roy