osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [BucketingSink] notify on moving into pending/ final state


Hi Piotr, during the migration to the latest Flink version, we’ve decided to try to contribute this functionality to the master branch.

More details about hooking the state changes in BucketingSink are available in https://issues.apache.org/jira/browse/FLINK-9592 

Thx !

On 14 Jun 2018, at 23:29, Rinat <r.sharipov@xxxxxxxxxxxxx> wrote:

Hi Piotr, I’ve create an issue https://issues.apache.org/jira/browse/FLINK-9592

The third proposal looks great, may I try to contribute this issue ?

On 14 Jun 2018, at 12:29, Piotr Nowojski <piotr@xxxxxxxxxxxxxxxxx> wrote:

Hi,

Couple of things:

1. Please create a Jira ticket with this proposal, so we can move discussion from user mailing list.

I haven’t thought it through, so take my comments with a grain of salt, however:

2. If we were to go with such callback, I would prefer to have one BucketStateChangeCallback, with methods like `onInProgressToPending(…)`, `onPendingToFinal`, `onPendingToCancelled(…)`, etc, in oppose to having one interface passed three times/four times for different purposes.

3. Other thing that I had in mind is that BucketingSink could be rewritten to extend TwoPhaseCommitSinkFunction. In that case, with 

public class BucketingSink2 extends TwoPhaseCommitSinkFunction<???>

user could add his own hooks by overriding following methods

BucketingSink2#beginTransaction, BucketingSink2#preCommit, BucketingSink2#commit, BucketingSink2#abort. For example:

public class MyBucketingSink extends BucketingSink2 {
  @Override
  protected void  commit(??? txn) {
    super.commit(txn);
    // My hook on moving file from pending to commit state
  };
}

Alternatively, we could implement before mentioned callbacks support in TwoPhaseCommitSinkFunction and provide such feature to Kafka/Pravega/BucketingSink at once.

Piotrek

Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team

mobile: +7 (925) 416-37-26

CleverDATA
make your data clever


Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team

mobile: +7 (925) 416-37-26

CleverDATA
make your data clever