osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Proposal] Utilities for reading, transforming and creatingStreaming savepoints


Hi,

also +1.
As vivo said, SavePoint is not compatible.
I have heard from a lot of users just said "my previous program does not
work any more!".
If these utilities provide such migration functions, it would be perfect!

Best,
tison.


vino yang <yanghua1127@xxxxxxxxx> 于2018年8月18日周六 上午11:04写道:

> Hi,
>
> +1, from my side. Considering that Savepoint is not compatible with every
> version of Flink.
> It would be very useful if it could do some conversion between different
> versions of Flink Savepoint.
>
> Thanks, vino.
>
> bupt_ljy <bupt_ljy@xxxxxxx> 于2018年8月18日周六 上午2:55写道:
>
> > Hi,
> > +1, I think it will be a very great tool for Flink, especially the
> > creating new state part. On production, we’re really worried about the
> > availability of the savepoints, because the generating logic is inside
> > Flink and we don’t have a good way to validate it. But with this tool, we
> > can construct a new state for our programs very soon even if the
> savepoints
> > data is broken.
> > It’s great, thanks!
> >
> >
> > Original Message
> > Sender:Jamie Grierjgrier@xxxxxxxx
> > Recipient:devdev@xxxxxxxxxxxxxxxx
> > Date:Saturday, Aug 18, 2018 02:32
> > Subject:Re: [Proposal] Utilities for reading, transforming and
> > creatingStreaming savepoints
> >
> >
> > This is great, Gyula! A colleague here at Lyft has also done some work
> > around bootstrapping DataStream programs and we've also talked a bit
> about
> > doing this by running DataSet programs. On Fri, Aug 17, 2018 at 3:28 AM,
> > Gyula Fóra gyula.fora@xxxxxxxxx wrote:  Hi All!   I want to share with
> > you a little project we have been working on at King  (with some help
> from
> > some dataArtisans folks). I think this would be a  valuable addition to
> > Flink and solve a bunch of outstanding production  use-cases and
> headaches
> > around state bootstrapping and state analytics.   We have built a quick
> and
> > dirty POC implementation on top of Flink 1.6,  please check the README
> for
> > some nice examples to get a quick idea:   https://github.com/king/bravo
> >  *Short story*  Bravo is a convenient state reader and writer library
> > leveraging the  Flink’s batch processing capabilities. It supports
> > processing and writing  Flink streaming savepoints. At the moment it only
> > supports processing  RocksDB savepoints but this can be extended in the
> > future for other state  backends and checkpoint types.   Our goal is to
> > cover a few basic features:   - Converting keyed states to Flink DataSets
> > for processing and analytics  - Reading/Writing non-keyed operators
> states
> > - Bootstrap keyed states from Flink DataSets and create new valid
> > savepoints  - Transform existing savepoints by replacing/changing some
> > states    Some example use-cases:   - Point-in-time state analytics
> across
> > all operators and keys  - Bootstrap state of a streaming job from
> external
> > resources such as  reading from database/filesystem  - Validate and
> > potentially repair corrupted state of a streaming job  - Change max
> > parallelism of a job    Our main goal is to start working together with
> > other Flink production  users and make this something useful that can be
> > part of Flink. So if you  have use-cases please talk to us :)  I have
> also
> > started a google doc which contains a little bit more info than  the
> readme
> > and could be a starting place for discussions:
> > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpw
> > dhqBMr-ppkFL5E/edit?usp=sharing   I know there are a bunch of rough edges
> > and bugs (and no tests) but our  motto is: If you are not embarrassed,
> you
> > released too late :)   Please let me know what you think!   Cheers,
> Gyula
>