[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Apache Flink CEP how to detect if event did not occur within x seconds?

Hi Florin,

I think Dawid might help you. I am pulling him into the discussion.


On 12 Dec 2018, at 16:24, Spico Florin <spicoflorin@xxxxxxxxx> wrote:

I'm using the same questions as in this stackoverflow post https://stackoverflow.com/questions/50356981/apache-flink-cep-how-to-detect-if-event-did-not-occur-within-x-seconds?rq=1, due to fact that I need the same functionality.

My use case is to detect when an device is disconnected from a server (for example).
From SNMP I'm sending the data to Kafka then  to Flink CEP and check whether I have the established connection within X seconds.
If the window of X seconds timedout, then I can consider that the device is disconnected.
The issue is that, when the device is disconnected, SNMP does't send anymore data about that port that was used. Thus no event is sent to CEP. Only when the device is connected again then, the disconnected event is triggered.

I checked the status of the jira ticket regarding this feature:

but unfortunately, is still open.

In the official documentation, https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html, in the paragraph related with Idling sources, somehow the situation described resembles what we need:
"Currently, with pure event time watermarks generators, watermarks can not progress if there are no elements to be processed. That means in case of gap in the 
incoming data, event time will not progress and for example the window operator will not be triggered and thus existing windows will not be able to produce any output data.
To circumvent this one can use periodic watermark assigners that don’t only assign based on element timestamps. An example solution could be 
an assigner that switches to using current processing time as the time basis after not observing new events for a while."

Unfortunately, no example of how the mentioned assigner looks like. A full example will help us to check whether the mentioned solution works or not.

Therefore, can you please share your thoughts about this feature? 
Is feasible? If yes,please provide the Flink version that solve it and the example.
Is still a limitation and there is no plan to solve it in near future?

I look forward for your answers.

Best regards,