[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re:Re: Re: Re: stream failed when bootstrap

Only do rolling restart can't solve problem. But I thought I found a way.

There are two same name folder with different suffix in data dictionary. e.g. dayu_123 and dayu_234, the dayu_123 folder is empty and dayu_234 folder is not.
then I use cql to query system_schema.tables,the id of table named 'dayu' is 123。
because the data in table 'dayu' can be re-generate. I simplly remove dayu_234 folder when rolling restart.

Now when running nodetool resetlocalschema, there is no error. and I am bootstrap new node again. I will see is there any other problem left.

I hope my experience can help others.

Thanks for all your help!

At 2018-06-28 13:53:16, "kurt greaves" <kurt@xxxxxxxxxxxxxxx> wrote:
Yeah, but you only really need to drain, restart Cassandra one by one. Not that the others will hurt, but they aren't strictly necessary. 

On 28 June 2018 at 05:38, dayu <sdycreate@xxxxxxx> wrote:
Hi kurt, a rolling restart means run disablebinary, disablethrift, disablegossip, drain, stop cassandra and start cassandra command one by one, right?
Only one node is executed at a time


At 2018-06-28 11:37:43, "kurt greaves" <kurt@xxxxxxxxxxxxxxx> wrote:
Best off trying a rolling restart.

On 28 June 2018 at 03:18, dayu <sdycreate@xxxxxxx> wrote:
the output of nodetool describecluster
Cluster Information:
Name: online-xxx
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
c3f00d61-1ad7-3702-8703-af2a29e401c1: []

0568e8c1-48ba-3fb0-bb3c-462438978d7b: [, ....]

after I run nodetool resetlocalschema, a error log outcome

ERROR [InternalResponseStage:209417] 2018-06-28 11:14:12,904 MigrationTask.java:96 - Configuration
exception merging remote schema
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5552bba0-2
dc6-11e8-9b5c-254242d97235; expected 53f6d520-2dc6-11e8-948d-ab7caa3c8c36)
        at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790) ~[apac
        at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750) ~[apache-cassandra-3.0
        at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[apache-cassandra-3.0.1
        at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1348) ~[ap

At 2018-06-28 10:01:52, "Jeff Jirsa" <jjirsa@xxxxxxxxx> wrote:
You can sometimes bounce your way through it (or use nodetool resetlocalschema if it’s a single node that’s wrong), but there are some edge cases from which it’s very hard to recover

What’s the output of nodetool describecluster?

If you select from the schema tables, do you see that CFID on any real tables? 

Jeff Jirsa

On Jun 27, 2018, at 7:58 PM, dayu <sdycreate@xxxxxxx> wrote:

That sound reasonable, I have seen schema mismatch error before.
So any advise to deal with schema mismatches?

At 2018-06-28 09:50:37, "Jeff Jirsa" <jjirsa@xxxxxxxxx> wrote: >That log message says you did: > > CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming > >If you’re absolutely sure you didn’t, you should look for schema mismatches in your cluster > > >-- >Jeff Jirsa > > >> On Jun 27, 2018, at 7:49 PM, dayu <sdycreate@xxxxxxx> wrote: >> >> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming > >--------------------------------------------------------------------- >To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxche.org >For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx