[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] FLIP-27: Refactor Source Interface


Thanks Aljoscha for starting this, it’s blocking quite a lot of other possible improvements. I have one proposal. Instead of having a method:

boolean advance() throws IOException;

I would replace it with

 * Return a future, which when completed means that source has more data and getNext() will not block.
 * If you wish to use benefits of non blocking connectors, please implement this method appropriately.
default CompletableFuture<?> isBlocked() {
	return CompletableFuture.completedFuture(null);

And rename `getCurrent()` to `getNext()`.

Couple of arguments:
1. I don’t understand the division of work between `advance()` and `getCurrent()`. What should be done in which, especially for connectors that handle records in batches (like Kafka) and when should you call `advance` and when `getCurrent()`. 
2. Replacing `boolean` with `CompletableFuture<?>` will allow us in the future to have asynchronous/non blocking connectors and more efficiently handle large number of blocked threads, without busy waiting. While at the same time it doesn’t add much complexity, since naive connector implementations can be always blocking.
3. This also would allow us to use a fixed size thread pool of task executors, instead of one thread per task.


> On 31 Oct 2018, at 17:22, Aljoscha Krettek <aljoscha@xxxxxxxxxx> wrote:
> Hi All,
> In order to finally get the ball rolling on the new source interface that we have discussed for so long I finally created a FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> I cc'ed Thomas and Jamie because of the ongoing work/discussion about adding per-partition watermark support to the Kinesis source and because this would enable generic implementation of event-time alignment for all sources. Maybe we need another FLIP for the event-time alignment part, especially the part about information sharing between operations (I'm not calling it state sharing because state has a special meaning in Flink).
> Please discuss away!
> Aljoscha