OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: coordinate watermarks between jobs?


Hi Tao,

The watermarks of operators that consume from two (or more) streams are always synced to the lowest watermark.
This behavior guarantees that data won't be late (unless it was late when watermarks were assigned). However, the operator will most likely need to buffer more events from the "faster" streams.

Right now, it is not possible to throttle faster streams to the pace of the slowest stream.

Best, Fabian

2018-04-27 1:05 GMT+02:00 Tao Xia <tao@xxxxxxxxxxx>:
Hi All,
  I am trying to reply events from 3 different sources and hopefully in time sequence, say Stream1, Stream2, Stream3. Since their size vary a lot, the watermarks on one stream is much faster than other streams.  Is there any way to coordinate the watermarks between different input streams. 
Thanks,
Tao