[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Window state with rocksdb backend

Hi all,

This is mingliang, I got a problem with rocksdb backend.

I'm currently using a 15min SessionWindow which also fires every 10s, there's no pre-aggregation, so the input of WindowFunction would be the whole Iterator of input object.

For window operator, I assume this collection is also a state that maintained by Flink.

Then, in each 10s fire, the window function will take the objects out from iterator and do some update, and in next fire, I assume I would get the updated value of that object.

With File system backend it was successful but eats a lot of memory and finally I got GC overhead limit, then I switch to rocksdb backend and the problem is the object in the next fire round is not updated by the previous fire round.

Do I have to do some additional staff with rocksdb backend in this case?

Thanks in advance


This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.