[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cassandra 2.1 bootstrap - No streaming progress from one node


+1 with Sean above.
In Cassandra 2.2 you got the new 'nodetool resume' command to resume a bootstrap. For C*2.1, nothing equivalent sadly. Thus old school technics apply and the possible alternatives are

Option 1 - safe and slow- Stop Cassandra from the stuck joining node. Remove everything (comitlog/data), restart bootstrap, mentioned by Sean above.
Option 2 - More wild / hopefully quicker - Stop the joining node and start it with 'auto_bootstrap: false'. The node joins with missing data, that you can repair after. Yet this can be done without inconsistencies only using a strong consistency (CL.R+CL.W > RF). This node will be read from but is then not enough to induce a stale read.

Unless you're confident or have a huge amount of data and most of it made it to the new node already, I would stick with the option 1, safe and slow (and upgrade soon ;-))

Alain Rodriguez - alain@xxxxxxxxxxxxxxxxx
France / Spain

The Last Pickle - Apache Cassandra Consulting

Le mer. 7 nov. 2018 à 19:22, Durity, Sean R <SEAN_R_DURITY@xxxxxxxxxxxxx> a écrit :

I would wipe the new node and bootstrap again. I do not know of any way to resume the streaming that was previously in progress.



Sean Durity

From: Steinmaurer, Thomas <thomas.steinmaurer@xxxxxxxxxxxxx>
Sent: Wednesday, November 07, 2018 5:13 AM
To: user@xxxxxxxxxxxxxxxxxxxx
Subject: [EXTERNAL] Cassandra 2.1 bootstrap - No streaming progress from one node




while bootstrapping a new node into an existing cluster, a node which is acting as source for streaming got restarted unfortunately. Since then, from nodetool netstats I don’t see any progress for this particular node anymore.





        Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 GB total


Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our case) not doing any resume here, in case there have been e.g. connectivity troubles or in our case, Cassandra on the node acting as stream source got restarted?


Can I force the joining node to recover connection to X.X.X.X or do I need to restart the bootstrap via restart on the new node from scratch?





The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.