osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Portable streaming side inputs


I'm working on support for side inputs in streaming mode in the portable Flink runner [1].

The runner would be responsible for holding main inputs (durably) when side inputs are not available.

To check if side inputs are available, the window mapping function is required (see SimplePushbackSideInputDoFnRunner.isReady).

The window mapping function (like viewFn) is SDK specific serialized in the proto (see PCollectionViewTranslation). 

With Python SDK, the Flink runner cannot rehydrate these objects.

Is my understanding correct and what assumptions can be made about the window mapping in the runner?

Thanks,
Thomas

[1] https://issues.apache.org/jira/browse/BEAM-2930