OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Repair scheduling tools


Schedule scheme looks good. I believe in process / sidecar can both coexist. As an admin would love to be able to run one or the other or none.

Thank you for taking a lead and producing a plan that can actually be executed.

--
Rahul Singh
rahul.singh@xxxxxxxx

Anant Corporation

On Apr 12, 2018, 6:35 PM -0400, Joseph Lynch <joe.e.lynch@xxxxxxxxx>, wrote:
> Given the feedback here and on the ticket, I've written up a proposal
> for a repair
> sidecar tool
> <https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#heading=h.5f10ng8gzle8
> in the ticket's design document. If there are no major concerns we're going
> to start working on porting the Priam implementation into this new tool
> soon.
>
> -Joey
>
> On Tue, Apr 10, 2018 at 4:17 PM, Elliott Sims <elliott@xxxxxxxxxxxxx> wrote:
>
> > My two cents as a (relatively small) user. I'm coming at this from the
> > ops/user side, so my apologies if some of these don't make sense based on a
> > more detailed understanding of the codebase:
> >
> > Repair is definitely a major missing piece of Cassandra. Integrated would
> > be easier, but a sidecar might be more flexible. As an intermediate step
> > that works towards both options, does it make sense to start with
> > finer-grained tracking and reporting for subrange repairs? That is, expose
> > a set of interfaces (both internally and via JMX) that give a scheduler
> > enough information to run subrange repairs across multiple keyspaces or
> > even non-overlapping ranges at the same time. That lets people experiment
> > with and quickly/safely/easily iterate on different scheduling strategies
> > in the short term, and long-term those strategies can be integrated into a
> > built-in scheduler
> >
> > On the subject of scheduling, I think adjusting parallelism/aggression with
> > a possible whitelist or blacklist would be a lot more useful than a "time
> > between repairs". That is, if repairs run for a few hours then don't run
> > for a few (somewhat hard-to-predict) hours, I still have to size the
> > cluster for the load when the repairs are running. The only reason I can
> > think of for an interval between repairs is to allow re-compaction from
> > repair anticompactions, and subrange repairs seem to eliminate this. Even
> > if they didn't, a more direct method along the lines of "don't repair when
> > the compaction queue is too long" might make more sense. Blacklisted
> > timeslots might be useful for avoiding peak time or batch jobs, but only if
> > they can be specified for consistent time-of-day intervals instead of
> > unpredictable lulls between repairs.
> >
> > I really like the idea of automatically adjusting gc_grace_seconds based on
> > repair state. The only_purge_repaired_tombstones option fixes this
> > elegantly for sequential/incremental repairs on STCS, but not for subrange
> > repairs or LCS (unless a scheduler gains the ability somehow to determine
> > that every subrange in an sstable has been repaired and mark it
> > accordingly?)
> >
> >
> > On 2018/04/03 17:48:14, Blake Eggleston <b...@xxxxxxxxx> wrote:
> > > Hi dev@,
> > >
> > > >
> > >
> > > The question of the best way to schedule repairs came up on
> > CASSANDRA-14346, and I thought it would be good to bring up the idea of an
> > external tool on the dev list.
> > >
> > > >
> > >
> > > Cassandra lacks any sort of tools for automating routine tasks that are
> > required for running clusters, specifically repair. Regular repair is a
> > must for most clusters, like compaction. This means that, especially as far
> > as eventual consistency is concerned, Cassandra isn’t totally functional
> > out of the box. Operators either need to find a 3rd party solution or
> > implement one themselves. Adding this to Cassandra would make it easier to
> > use.
> > >
> > > >
> > >
> > > Is this something we should be doing? If so, what should it look like?
> > >
> > > >
> > >
> > > Personally, I feel like this is a pretty big gap in the project and would
> > like to see an out of process tool offered. Ideally, Cassandra would just
> > take care of itself, but writing a distributed repair scheduler that you
> > trust to run in production is a lot harder than writing a single process
> > management application that can failover.
> > >
> > > >
> > >
> > > Any thoughts on this?
> > >
> > > >
> > >
> > > Thanks,
> > >
> > > >
> > >
> > > Blake
> > >
> > >
> >