osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] changing default token behavior for 4.0


> We should create a JIRA to find what other defaults we need revisit.

Changing a default is a pretty big deal, I think we should discuss any
changes to defaults here on the ML before moving it into JIRA.  It's nice
to get a bit more discussion around the change than what happens in JIRA.

We (TLP) did some testing on 4 tokens and found it to work surprisingly
well.   It wasn't particularly formal, but we verified the load stays
pretty even with only 4 tokens as we added nodes to the cluster.  Higher
token count hurts availability by increasing the number of nodes any given
node is a neighbor with, meaning any 2 nodes that fail have an increased
chance of downtime when using QUORUM.  In addition, with the recent
streaming optimization it seems the token counts will give a greater chance
of a node streaming entire sstables (with LCS), meaning we'll do a better
job with node density out of the box.

Next week I can try to put together something a little more convincing.
Weekend time.

Jon


On Fri, Sep 21, 2018 at 8:45 PM sankalp kohli <kohlisankalp@xxxxxxxxx>
wrote:

> +1 to lowering it.
> Thanks Jon for starting this.We should create a JIRA to find what other
> defaults we need revisit. (Please keep this discussion for "default token"
> only.  )
>
> On Fri, Sep 21, 2018 at 8:26 PM Jeff Jirsa <jjirsa@xxxxxxxxx> wrote:
>
> > Also agree it should be lowered, but definitely not to 1, and probably
> > something closer to 32 than 4.
> >
> > --
> > Jeff Jirsa
> >
> >
> > > On Sep 21, 2018, at 8:24 PM, Jeremy Hanna <jeremy.hanna1234@xxxxxxxxx>
> > wrote:
> > >
> > > I agree that it should be lowered. What I’ve seen debated a bit in the
> > past is the number but I don’t think anyone thinks that it should remain
> > 256.
> > >
> > >> On Sep 21, 2018, at 7:05 PM, Jonathan Haddad <jon@xxxxxxxxxxxxx>
> wrote:
> > >>
> > >> One thing that's really, really bothered me for a while is how we
> > default
> > >> to 256 tokens still.  There's no experienced operator that leaves it
> as
> > is
> > >> at this point, meaning the only people using 256 are the poor folks
> that
> > >> just got started using C*.  I've worked with over a hundred clusters
> in
> > the
> > >> last couple years, and I think I only worked with one that had lowered
> > it
> > >> to something else.
> > >>
> > >> I think it's time we changed the default to 4 (or 8, up for debate).
> > >>
> > >> To improve the behavior, we need to change a couple other things.  The
> > >> allocate_tokens_for_keyspace setting is... odd.  It requires you have
> a
> > >> keyspace already created, which doesn't help on new clusters.  What
> I'd
> > >> like to do is add a new setting, allocate_tokens_for_rf, and set it to
> > 3 by
> > >> default.
> > >>
> > >> To handle clusters that are already using 256 tokens, we could prevent
> > the
> > >> new node from joining unless a -D flag is set to explicitly allow
> > >> imbalanced tokens.
> > >>
> > >> We've agreed to a trunk freeze, but I feel like this is important
> enough
> > >> (and pretty trivial) to do now.  I'd also personally characterize this
> > as a
> > >> bug fix since 256 is horribly broken when the cluster gets to any
> > >> reasonable size, but maybe I'm alone there.
> > >>
> > >> I honestly can't think of a use case where random tokens is a good
> > choice
> > >> anymore, so I'd be fine / ecstatic with removing it completely and
> > >> requiring either allocate_tokens_for_keyspace (for existing clusters)
> > >> or allocate_tokens_for_rf
> > >> to be set.
> > >>
> > >> Thoughts?  Objections?
> > >> --
> > >> Jon Haddad
> > >> http://www.rustyrazorblade.com
> > >> twitter: rustyrazorblade
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> > > For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> > For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade