[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Better way to clean up state when connect

Hi there,

I have a use case to check for active ID, there are two streams and I connect them: one has actual data (Stream A) and the other one is for lookup purpose (Stream B), I am getting Stream B as a file which includes all active ID, so inactive ID would not be show up on this list. I tried to use watermark to clean up the state of inactivate ID, but the Stream B updates is unpredictable so I want to keep everything in state until I found the item is not in that file any more. 

Please suggest what is the best way to implement it in flink. Thanks in advance for your help.