osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] changing default token behavior for 4.0


Jon, thanks for starting this thread!

I have created CASSANDRA-14784 to track this. 

Dinesh

> On Sep 21, 2018, at 9:18 PM, Sankalp Kohli <kohlisankalp@xxxxxxxxx> wrote:
> 
> Putting it on JIRA is to make sure someone is assigned to it and it is tracked. Changes should be discussed over ML like you are saying. 
> 
> On Sep 21, 2018, at 21:02, Jonathan Haddad <jon@xxxxxxxxxxxxx> wrote:
> 
>>> We should create a JIRA to find what other defaults we need revisit.
>> 
>> Changing a default is a pretty big deal, I think we should discuss any
>> changes to defaults here on the ML before moving it into JIRA.  It's nice
>> to get a bit more discussion around the change than what happens in JIRA.
>> 
>> We (TLP) did some testing on 4 tokens and found it to work surprisingly
>> well.   It wasn't particularly formal, but we verified the load stays
>> pretty even with only 4 tokens as we added nodes to the cluster.  Higher
>> token count hurts availability by increasing the number of nodes any given
>> node is a neighbor with, meaning any 2 nodes that fail have an increased
>> chance of downtime when using QUORUM.  In addition, with the recent
>> streaming optimization it seems the token counts will give a greater chance
>> of a node streaming entire sstables (with LCS), meaning we'll do a better
>> job with node density out of the box.
>> 
>> Next week I can try to put together something a little more convincing.
>> Weekend time.
>> 
>> Jon
>> 
>> 
>> On Fri, Sep 21, 2018 at 8:45 PM sankalp kohli <kohlisankalp@xxxxxxxxx>
>> wrote:
>> 
>>> +1 to lowering it.
>>> Thanks Jon for starting this.We should create a JIRA to find what other
>>> defaults we need revisit. (Please keep this discussion for "default token"
>>> only.  )
>>> 
>>>> On Fri, Sep 21, 2018 at 8:26 PM Jeff Jirsa <jjirsa@xxxxxxxxx> wrote:
>>>> 
>>>> Also agree it should be lowered, but definitely not to 1, and probably
>>>> something closer to 32 than 4.
>>>> 
>>>> --
>>>> Jeff Jirsa
>>>> 
>>>> 
>>>>> On Sep 21, 2018, at 8:24 PM, Jeremy Hanna <jeremy.hanna1234@xxxxxxxxx>
>>>> wrote:
>>>>> 
>>>>> I agree that it should be lowered. What I’ve seen debated a bit in the
>>>> past is the number but I don’t think anyone thinks that it should remain
>>>> 256.
>>>>> 
>>>>>> On Sep 21, 2018, at 7:05 PM, Jonathan Haddad <jon@xxxxxxxxxxxxx>
>>> wrote:
>>>>>> 
>>>>>> One thing that's really, really bothered me for a while is how we
>>>> default
>>>>>> to 256 tokens still.  There's no experienced operator that leaves it
>>> as
>>>> is
>>>>>> at this point, meaning the only people using 256 are the poor folks
>>> that
>>>>>> just got started using C*.  I've worked with over a hundred clusters
>>> in
>>>> the
>>>>>> last couple years, and I think I only worked with one that had lowered
>>>> it
>>>>>> to something else.
>>>>>> 
>>>>>> I think it's time we changed the default to 4 (or 8, up for debate).
>>>>>> 
>>>>>> To improve the behavior, we need to change a couple other things.  The
>>>>>> allocate_tokens_for_keyspace setting is... odd.  It requires you have
>>> a
>>>>>> keyspace already created, which doesn't help on new clusters.  What
>>> I'd
>>>>>> like to do is add a new setting, allocate_tokens_for_rf, and set it to
>>>> 3 by
>>>>>> default.
>>>>>> 
>>>>>> To handle clusters that are already using 256 tokens, we could prevent
>>>> the
>>>>>> new node from joining unless a -D flag is set to explicitly allow
>>>>>> imbalanced tokens.
>>>>>> 
>>>>>> We've agreed to a trunk freeze, but I feel like this is important
>>> enough
>>>>>> (and pretty trivial) to do now.  I'd also personally characterize this
>>>> as a
>>>>>> bug fix since 256 is horribly broken when the cluster gets to any
>>>>>> reasonable size, but maybe I'm alone there.
>>>>>> 
>>>>>> I honestly can't think of a use case where random tokens is a good
>>>> choice
>>>>>> anymore, so I'd be fine / ecstatic with removing it completely and
>>>>>> requiring either allocate_tokens_for_keyspace (for existing clusters)
>>>>>> or allocate_tokens_for_rf
>>>>>> to be set.
>>>>>> 
>>>>>> Thoughts?  Objections?
>>>>>> --
>>>>>> Jon Haddad
>>>>>> http://www.rustyrazorblade.com
>>>>>> twitter: rustyrazorblade
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
>>>>> For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
>>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
>>>> For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
>>>> 
>>>> 
>>> 
>> 
>> 
>> -- 
>> Jon Haddad
>> http://www.rustyrazorblade.com
>> twitter: rustyrazorblade
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx