OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Repair scheduling tools


Given the feedback here and on the ticket, I've written up a proposal
for a repair
sidecar tool
<https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#heading=h.5f10ng8gzle8>
in the ticket's design document. If there are no major concerns we're going
to start working on porting the Priam implementation into this new tool
soon.

-Joey

On Tue, Apr 10, 2018 at 4:17 PM, Elliott Sims <elliott@xxxxxxxxxxxxx> wrote:

> My two cents as a (relatively small) user.  I'm coming at this from the
> ops/user side, so my apologies if some of these don't make sense based on a
> more detailed understanding of the codebase:
>
> Repair is definitely a major missing piece of Cassandra.  Integrated would
> be easier, but a sidecar might be more flexible.  As an intermediate step
> that works towards both options, does it make sense to start with
> finer-grained tracking and reporting for subrange repairs?  That is, expose
> a set of interfaces (both internally and via JMX) that give a scheduler
> enough information to run subrange repairs across multiple keyspaces or
> even non-overlapping ranges at the same time.  That lets people experiment
> with and quickly/safely/easily iterate on different scheduling strategies
> in the short term, and long-term those strategies can be integrated into a
> built-in scheduler
>
> On the subject of scheduling, I think adjusting parallelism/aggression with
> a possible whitelist or blacklist would be a lot more useful than a "time
> between repairs".  That is, if repairs run for a few hours then don't run
> for a few (somewhat hard-to-predict) hours, I still have to size the
> cluster for the load when the repairs are running.   The only reason I can
> think of for an interval between repairs is to allow re-compaction from
> repair anticompactions, and subrange repairs seem to eliminate this.  Even
> if they didn't, a more direct method along the lines of "don't repair when
> the compaction queue is too long" might make more sense.  Blacklisted
> timeslots might be useful for avoiding peak time or batch jobs, but only if
> they can be specified for consistent time-of-day intervals instead of
> unpredictable lulls between repairs.
>
> I really like the idea of automatically adjusting gc_grace_seconds based on
> repair state.  The only_purge_repaired_tombstones option fixes this
> elegantly for sequential/incremental repairs on STCS, but not for subrange
> repairs or LCS (unless a scheduler gains the ability somehow to determine
> that every subrange in an sstable has been repaired and mark it
> accordingly?)
>
>
> On 2018/04/03 17:48:14, Blake Eggleston <b...@xxxxxxxxx> wrote:
> > Hi dev@,>
> >
> >  >
> >
> > The question of the best way to schedule repairs came up on
> CASSANDRA-14346, and I thought it would be good to bring up the idea of an
> external tool on the dev list.>
> >
> >  >
> >
> > Cassandra lacks any sort of tools for automating routine tasks that are
> required for running clusters, specifically repair. Regular repair is a
> must for most clusters, like compaction. This means that, especially as far
> as eventual consistency is concerned, Cassandra isn’t totally functional
> out of the box. Operators either need to find a 3rd party solution or
> implement one themselves. Adding this to Cassandra would make it easier to
> use.>
> >
> >  >
> >
> > Is this something we should be doing? If so, what should it look like?>
> >
> >  >
> >
> > Personally, I feel like this is a pretty big gap in the project and would
> like to see an out of process tool offered. Ideally, Cassandra would just
> take care of itself, but writing a distributed repair scheduler that you
> trust to run in production is a lot harder than writing a single process
> management application that can failover.>
> >
> >  >
> >
> > Any thoughts on this?>
> >
> >  >
> >
> > Thanks,>
> >
> >  >
> >
> > Blake>
> >
> >
>