OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

java.lang.AssertionError: Memory was freed during index rebuild


Has anyone seen this error or might have an idea what is causing this?

Version: 3.5

java.lang.AssertionError: Memory was freed
        at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260)
        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
        at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
        at org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
        at org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:391)
        at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeOne(ClusteringPrefix.java:480)
        at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeAll(ClusteringPrefix.java:486)
        at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeNextClustering(ClusteringPrefix.java:502)
        at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:210)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
        at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
        at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
        at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
        at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

I ran `nodetool rebuild_index -- keyspace table index` on 6 nodes of a 6 node cluster. After several hours of processing, it failed on 3 of the nodes with the following additional call stacks:

java.lang.AssertionError: Memory was freed
        at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260)
        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
        at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
        at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
        at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63)
        at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
        at org.apache.cassandra.db.Columns$Serializer.deserializeSubset(Columns.java:494)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:429)
        at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
        at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
        at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
        at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
        at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

java.lang.AssertionError: Memory was freed
        at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260)
        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
        at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
        at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
        at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:73)
        at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
        at org.apache.cassandra.db.SerializationHeader.readTimestamp(SerializationHeader.java:205)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:420)
        at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
        at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
        at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
        at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
        at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Is something timing out and freeing up the memory?
Mark Bryant
[Sumo-Logo]     [Tel] +44 (0)114 2426766
[Web]www.sumo-digital.com
[Address]32 Jessops Riverside, 800 Brightside Lane, Sheffield, S9 2RX, United Kingdom
[------------------------------------------------------------------------------------------------------------------------------]
[Facebook]<https://www.facebook.com/SumoDigital> [Twitter] <https://twitter.com/sumodigitalltd>  [LinkedIn] <https://www.linkedin.com/company/sumo-digital>
The contents of this email are confidential and may not be copied, disclosed to other parties or used by anyone other than the intended recipient. If you received this email in error, please contact Sumo Digital Ltd by email: info@xxxxxxxxxxxxxxxx or by calling +44 (0)114 2426766. Quote the name of the sender, your email address and then delete this email from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx