[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [BucketingSink] notify on moving into pending/ final state

Hi Rinat,

I have commented on your PR and on the JIRA.
Let me know what you think.


On Oct 11, 2018, at 4:45 PM, Dawid Wysakowicz <dwysakowicz@xxxxxxxxxx> wrote:

Hi Ribat, 
I haven't checked your PR but we introduced a new connector in flink 1.6 called StreamingFileSink that is supposed to replace BucketingSink long term. I think it might solve a few problems of yours. Have you checked it by chance?


On Thu, 11 Oct 2018, 14:10 Rinat, <r.sharipov@xxxxxxxxxxxxx> wrote:
Hi Piotr, during the migration to the latest Flink version, we’ve decided to try to contribute this functionality to the master branch.

More details about hooking the state changes in BucketingSink are available in https://issues.apache.org/jira/browse/FLINK-9592 

Thx !

On 14 Jun 2018, at 23:29, Rinat <r.sharipov@xxxxxxxxxxxxx> wrote:

Hi Piotr, I’ve create an issue https://issues.apache.org/jira/browse/FLINK-9592

The third proposal looks great, may I try to contribute this issue ?

On 14 Jun 2018, at 12:29, Piotr Nowojski <piotr@xxxxxxxxxxxxxxxxx> wrote:


Couple of things:

1. Please create a Jira ticket with this proposal, so we can move discussion from user mailing list.

I haven’t thought it through, so take my comments with a grain of salt, however:

2. If we were to go with such callback, I would prefer to have one BucketStateChangeCallback, with methods like `onInProgressToPending(…)`, `onPendingToFinal`, `onPendingToCancelled(…)`, etc, in oppose to having one interface passed three times/four times for different purposes.

3. Other thing that I had in mind is that BucketingSink could be rewritten to extend TwoPhaseCommitSinkFunction. In that case, with 

public class BucketingSink2 extends TwoPhaseCommitSinkFunction<???>

user could add his own hooks by overriding following methods

BucketingSink2#beginTransaction, BucketingSink2#preCommit, BucketingSink2#commit, BucketingSink2#abort. For example:

public class MyBucketingSink extends BucketingSink2 {
  protected void  commit(??? txn) {
    // My hook on moving file from pending to commit state

Alternatively, we could implement before mentioned callbacks support in TwoPhaseCommitSinkFunction and provide such feature to Kafka/Pravega/BucketingSink at once.


Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team

mobile: +7 (925) 416-37-26

make your data clever

Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team

mobile: +7 (925) 416-37-26

make your data clever