[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PROPOSAL] Improving Flink’s timer management for large state


> On 29. May 2018, at 09:34, Till Rohrmann <trohrmann@xxxxxxxxxx> wrote:
> Thanks for the great design document Stefan. Unifying how Flink handles
> state such that all checkpointable state is maintained by the StateBackend
> makes a lot of sense to me. Also making the timer service scalable and
> adding support for asynchronous checkpoints is really important for many
> Flink users. Consequently +1 for this improvement.
> Cheers,
> Till
> On Mon, May 28, 2018 at 9:45 AM, Stefan Richter <s.richter@xxxxxxxxxxxxxxxxx
>> wrote:
>> Thanks for the positive feedback so far!
>> @Sihua: I totally agree with your comments about improvements in
>> performance for the existing RocksDB timer code. In fact, that is why I
>> phrased it like „ implementation that is loosely based on some ideas“ to
>> point out a solution can look roughly like the existing code but that it
>> probably should *not* be a simple copy-paste. As for the comment of
>> supporting heap timers with RocksDB state, I think there is nothing that
>> speaks fundamentally against it and I think the design can be in a way to
>> support that. It just makes the configuration more complex and we need to
>> slightly „special case“ in incremental checkpoints. I was already wondering
>> if heap timers with RocksDB state could not already become a byproduct of a
>> stepwise implementation, i.e. when the first step of the plan is pushing
>> timer state into the backends and there does not yet exist a RocksDB timer
>> state.
>>> Am 27.05.2018 um 10:03 schrieb sihua zhou <summerleafs@xxxxxxx>:
>>> I also +1 for this very good proposal!
>>> In general, the design is good, especially the part the related to the
>> timer on Heap, but refer to the part of the timer on RocksDB, I think there
>> may still exist some improvement that we can do, I just left the comments
>> on the doc.
>>> Best, Sihua
>>> On 05/27/2018 15:31,Bowen Li<bowenli86@xxxxxxxxx> wrote:
>>> +1 LGTM. RocksDB timer service is one of the most highly anticipated
>>> features from Flink users, and it's finally coming, officially. I also
>>> would love to see bringing timer more closely to state backend, for the
>>> sake of easier development and maintenance of code.
>>> On Fri, May 25, 2018 at 7:13 AM, Stefan Richter <
>> s.richter@xxxxxxxxxxxxxxxxx
>>> wrote:
>>> Hi,
>>> I am currently planning how to improve Flink’s timer management for large
>>> state. In particular, I would like to introduce timer state that is
>> managed
>>> in RocksDB and also to improve the capabilities of the heap-based timer
>>> service, e.g. support for asynchronous checkpoints. You can find a short
>>> outline of my planned approach in this document:
>>> https://docs.google.com/document/d/1XbhJRbig5c5Ftd77d0mKND1bePyTC
>>> 26Pz04EvxdA7Jc/edit?usp=sharing
>>> As always, your questions, feedback, and comments are highly appreciated.
>>> Best,
>>> Stefan