osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Window state with rocksdb backend


Hi all,


This is mingliang, I got a problem with rocksdb backend.


I'm currently using a 15min SessionWindow which also fires every 10s, there's no pre-aggregation, so the input of WindowFunction would be the whole Iterator of input object.

For window operator, I assume this collection is also a state that maintained by Flink.

Then, in each 10s fire, the window function will take the objects out from iterator and do some update, and in next fire, I assume I would get the updated value of that object.

With File system backend it was successful but eats a lot of memory and finally I got GC overhead limit, then I switch to rocksdb backend and the problem is the object in the next fire round is not updated by the previous fire round.

Do I have to do some additional staff with rocksdb backend in this case?


Thanks in advance

Mingliang



本邮件及其附件含有小红书公司的保密信息,仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件!
This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.