OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Cassandra crashed during major compaction


After a bulk load of writes to existing partition keys (with a higher timestamp), I wanted to free disk space, suspecting that rows will be in the highest levels and it would take a time until they were compacted.
I've started a major compaction, and the disk usage went from ~30% to ~40% (as expected) but after ~10 hs Cassandra has crashed (*) and disk usage continues being ~40%, even after restart.

How could I remove SSTables created during the failed compaction?

(*) From the logs, I understand that the JVM runs out of java heap space. However, I don't understand why there are multiple OutOfMemory errors, it shouldn't crashed with the first one?

ERROR [MessagingService-Incoming-/x.x.x.x] 2018-11-06 22:28:35,251 CassandraDaemon.java:207 - Exception in thread Thread[MessagingService-Incoming-/x.x.x.x,5,main]
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:403) ~[apache-cassandra-3.0.13.jar:3.0.13]
at org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:341) ~[apache-cassandra-3.0.13.jar:3.0.13]
at org.apache.cassandra.db.ReadResponse$Serializer.deserialize(ReadResponse.java:382) ~[apache-cassandra-3.0.13.jar:3.0.13]
--
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.0.13.jar:3.0.13]
INFO [HintsDispatcher:63] 2018-11-06 22:34:49,678 HintsDispatchExecutor.java:271 - Finished hinted handoff of file 9584b93c-f86e-464f-a9ba-3dd33134b7af-1541550621736-1.hints to endpoint /y.y.y.y: 9584b93c-f86e-464f-a9ba-3dd33134b7af, partially
ERROR [MessagingService-Incoming-/z.z.z.z] 2018-11-06 22:37:11,099 CassandraDaemon.java:207 - Exception in thread Thread[MessagingService-Incoming-/z.z.z.z,5,main]
java.lang.OutOfMemoryError: Java heap space
INFO [ScheduledTasks:1] 2018-11-06 22:37:34,716 StatusLogger.java:56 - Sampler 0 0 0 0 0

ERROR [MessagingService-Incoming-/y.y.y.y] 2018-11-06 22:39:39,860 CassandraDaemon.java:207 - Exception in thread Thread[MessagingService-Incoming-/y.y.y.y,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/z.z.z.z] 2018-11-06 22:41:52,690 CassandraDaemon.java:207 - Exception in thread Thread[MessagingService-Incoming-/z.z.z.z,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/a.a.a.a] 2018-11-06 22:42:23,498 CassandraDaemon.java:207 - Exception in thread Thread[MessagingService-Incoming-/a.a.a.a,5,main]
java.lang.OutOfMemoryError: Java heap space
.....

ERROR [MessagingService-Incoming-/x.x.x.x] 2018-11-06 23:42:53,947 JVMStabilityInspector.java:140 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:403) ~[apache-cassandra-3.0.13.jar:3.0.13]
at org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:341) ~[apache-cassandra-3.0.13.jar:3.0.13]
at org.apache.cassandra.db.ReadResponse$Serializer.deserialize(ReadResponse.java:382) ~[apache-cassandra-3.0.13.jar:3.0.13]
--
at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.0.13.jar:3.0.13]
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.0.13.jar:3.0.13]
ERROR [SharedPool-Worker-6] 2018-11-06 23:42:53,947 JVMStabilityInspector.java:140 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-12] 2018-11-06 23:42:53,947 JVMStabilityInspector.java:140 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-18] 2018-11-06 23:42:53,948 JVMStabilityInspector.java:140 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space


Thanks.