OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] changing default token behavior for 4.0


We are using 8 or 16 tokens internally, with the token allocation algorithm
enabled. The range distribution is good for us.

Dikang.

On Fri, Sep 21, 2018 at 9:30 PM Dinesh Joshi <dinesh.joshi@xxxxxxxxx.invalid>
wrote:

> Jon, thanks for starting this thread!
>
> I have created CASSANDRA-14784 to track this.
>
> Dinesh
>
> > On Sep 21, 2018, at 9:18 PM, Sankalp Kohli <kohlisankalp@xxxxxxxxx>
> wrote:
> >
> > Putting it on JIRA is to make sure someone is assigned to it and it is
> tracked. Changes should be discussed over ML like you are saying.
> >
> > On Sep 21, 2018, at 21:02, Jonathan Haddad <jon@xxxxxxxxxxxxx> wrote:
> >
> >>> We should create a JIRA to find what other defaults we need revisit.
> >>
> >> Changing a default is a pretty big deal, I think we should discuss any
> >> changes to defaults here on the ML before moving it into JIRA.  It's
> nice
> >> to get a bit more discussion around the change than what happens in
> JIRA.
> >>
> >> We (TLP) did some testing on 4 tokens and found it to work surprisingly
> >> well.   It wasn't particularly formal, but we verified the load stays
> >> pretty even with only 4 tokens as we added nodes to the cluster.  Higher
> >> token count hurts availability by increasing the number of nodes any
> given
> >> node is a neighbor with, meaning any 2 nodes that fail have an increased
> >> chance of downtime when using QUORUM.  In addition, with the recent
> >> streaming optimization it seems the token counts will give a greater
> chance
> >> of a node streaming entire sstables (with LCS), meaning we'll do a
> better
> >> job with node density out of the box.
> >>
> >> Next week I can try to put together something a little more convincing.
> >> Weekend time.
> >>
> >> Jon
> >>
> >>
> >> On Fri, Sep 21, 2018 at 8:45 PM sankalp kohli <kohlisankalp@xxxxxxxxx>
> >> wrote:
> >>
> >>> +1 to lowering it.
> >>> Thanks Jon for starting this.We should create a JIRA to find what other
> >>> defaults we need revisit. (Please keep this discussion for "default
> token"
> >>> only.  )
> >>>
> >>>> On Fri, Sep 21, 2018 at 8:26 PM Jeff Jirsa <jjirsa@xxxxxxxxx> wrote:
> >>>>
> >>>> Also agree it should be lowered, but definitely not to 1, and probably
> >>>> something closer to 32 than 4.
> >>>>
> >>>> --
> >>>> Jeff Jirsa
> >>>>
> >>>>
> >>>>> On Sep 21, 2018, at 8:24 PM, Jeremy Hanna <
> jeremy.hanna1234@xxxxxxxxx>
> >>>> wrote:
> >>>>>
> >>>>> I agree that it should be lowered. What I’ve seen debated a bit in
> the
> >>>> past is the number but I don’t think anyone thinks that it should
> remain
> >>>> 256.
> >>>>>
> >>>>>> On Sep 21, 2018, at 7:05 PM, Jonathan Haddad <jon@xxxxxxxxxxxxx>
> >>> wrote:
> >>>>>>
> >>>>>> One thing that's really, really bothered me for a while is how we
> >>>> default
> >>>>>> to 256 tokens still.  There's no experienced operator that leaves it
> >>> as
> >>>> is
> >>>>>> at this point, meaning the only people using 256 are the poor folks
> >>> that
> >>>>>> just got started using C*.  I've worked with over a hundred clusters
> >>> in
> >>>> the
> >>>>>> last couple years, and I think I only worked with one that had
> lowered
> >>>> it
> >>>>>> to something else.
> >>>>>>
> >>>>>> I think it's time we changed the default to 4 (or 8, up for debate).
> >>>>>>
> >>>>>> To improve the behavior, we need to change a couple other things.
> The
> >>>>>> allocate_tokens_for_keyspace setting is... odd.  It requires you
> have
> >>> a
> >>>>>> keyspace already created, which doesn't help on new clusters.  What
> >>> I'd
> >>>>>> like to do is add a new setting, allocate_tokens_for_rf, and set it
> to
> >>>> 3 by
> >>>>>> default.
> >>>>>>
> >>>>>> To handle clusters that are already using 256 tokens, we could
> prevent
> >>>> the
> >>>>>> new node from joining unless a -D flag is set to explicitly allow
> >>>>>> imbalanced tokens.
> >>>>>>
> >>>>>> We've agreed to a trunk freeze, but I feel like this is important
> >>> enough
> >>>>>> (and pretty trivial) to do now.  I'd also personally characterize
> this
> >>>> as a
> >>>>>> bug fix since 256 is horribly broken when the cluster gets to any
> >>>>>> reasonable size, but maybe I'm alone there.
> >>>>>>
> >>>>>> I honestly can't think of a use case where random tokens is a good
> >>>> choice
> >>>>>> anymore, so I'd be fine / ecstatic with removing it completely and
> >>>>>> requiring either allocate_tokens_for_keyspace (for existing
> clusters)
> >>>>>> or allocate_tokens_for_rf
> >>>>>> to be set.
> >>>>>>
> >>>>>> Thoughts?  Objections?
> >>>>>> --
> >>>>>> Jon Haddad
> >>>>>> http://www.rustyrazorblade.com
> >>>>>> twitter: rustyrazorblade
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> >>>>> For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> >>>> For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
> >>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Jon Haddad
> >> http://www.rustyrazorblade.com
> >> twitter: rustyrazorblade
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> > For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
>
> --
Dikang