osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Watermark on keyed stream


Hi everyone,

 

it seems Flink only supports global watermarks currently which is a problem for my use case.

Many sensors send data which might be buffered for days in upstream systems before arriving at the Flink job.

The job keys the stream by sensor. If other sensors send values in the meantime, the global watermark is advanced

and buffered data that arrives late is dropped.

 

How could the issue be solved? I guess it would be possible to calculate the watermark manually and add it to a wrapper object,

but I am not sure how to correctly implement windowing (tumbling window) then.

 

Thank you in advance for any ideas.

 

Regards,

Nick