OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: java.lang.AssertionError: Memory was freed during index rebuild


3.5 is probably not a version you should be using in production in 2018 - it was a feature release and has had no bug fixes for years. Going up to 3.11.3 will likely fix many serious bugs you’re not noticing, and maybe the bug below you are noticing 


-- 
Jeff Jirsa


> On Oct 24, 2018, at 5:03 AM, Mark Bryant <MBryant@xxxxxxxxxxxxxxxx> wrote:
> 
> Has anyone seen this error or might have an idea what is causing this?
> 
> Version: 3.5
> 
> java.lang.AssertionError: Memory was freed
>        at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
>        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260)
>        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
>        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
>        at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
>        at org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400)
>        at org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
>        at org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:391)
>        at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeOne(ClusteringPrefix.java:480)
>        at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeAll(ClusteringPrefix.java:486)
>        at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeNextClustering(ClusteringPrefix.java:502)
>        at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:210)
>        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
>        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
>        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
>        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
>        at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
>        at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
>        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
>        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
>        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
>        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
>        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
>        at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
>        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
>        at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at java.lang.Thread.run(Thread.java:748)
> 
> I ran `nodetool rebuild_index -- keyspace table index` on 6 nodes of a 6 node cluster. After several hours of processing, it failed on 3 of the nodes with the following additional call stacks:
> 
> java.lang.AssertionError: Memory was freed
>        at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
>        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260)
>        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
>        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
>        at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
>        at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
>        at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63)
>        at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
>        at org.apache.cassandra.db.Columns$Serializer.deserializeSubset(Columns.java:494)
>        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:429)
>        at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
>        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
>        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
>        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
>        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
>        at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
>        at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
>        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
>        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
>        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
>        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
>        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
>        at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
>        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
>        at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at java.lang.Thread.run(Thread.java:748)
> 
> java.lang.AssertionError: Memory was freed
>        at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
>        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260)
>        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
>        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
>        at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
>        at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
>        at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:73)
>        at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
>        at org.apache.cassandra.db.SerializationHeader.readTimestamp(SerializationHeader.java:205)
>        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:420)
>        at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
>        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
>        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
>        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
>        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
>        at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
>        at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
>        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
>        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
>        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
>        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
>        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
>        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
>        at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
>        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
>        at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at java.lang.Thread.run(Thread.java:748)
> 
> Is something timing out and freeing up the memory?
> Mark Bryant
> [Sumo-Logo]     [Tel] +44 (0)114 2426766
> [Web]www.sumo-digital.com
> [Address]32 Jessops Riverside, 800 Brightside Lane, Sheffield, S9 2RX, United Kingdom
> [------------------------------------------------------------------------------------------------------------------------------]
> [Facebook]<https://www.facebook.com/SumoDigital> [Twitter] <https://twitter.com/sumodigitalltd>  [LinkedIn] <https://www.linkedin.com/company/sumo-digital>
> The contents of this email are confidential and may not be copied, disclosed to other parties or used by anyone other than the intended recipient. If you received this email in error, please contact Sumo Digital Ltd by email: info@xxxxxxxxxxxxxxxx or by calling +44 (0)114 2426766. Quote the name of the sender, your email address and then delete this email from your system.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx