OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Flink memory management in table api


Hi Amol,

The memory consumption depends on the query/operation that you are doing.
Time-based operations like group-window-aggregations,
over-window-aggregations, or window-joins can automatically clean up their
state once data is not no longer needed.
Operations such as non-windowed aggregations or joins have to persist all
data forever in state to guarantee absolute correctness.
However, you can also configure an idle state retention time [1] to remove
state that has not been accessed for a certain time.

Regarding you questions:

1) Unless you configure the idle state retention time, state is kept as
long as needed to guarantee correctness, potentially forever.
2) Queries use Flink's regular state features, i.e., you can configure the
RocksDBStateBackend to manage state on disk.
3) This depends on your query and the distribution of your data.

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/streaming.html#idle-state-retention-time

2018-07-04 7:46 GMT+02:00 Amol S - iProgrammer <amols@xxxxxxxxxxxxxxx>:

> Hello folks,
>
> I am using flink table api to join multiple tables and create a single
> table from them. I have some doubts in my mind.
>
> 1. How long the query will maintain partial results per key and how it
> maintains state of each key?
>
> 2. If it is maintains state in memory then the memory will continuously
> grows and it leads to memory over head.
>
> 3. How much RAM server  needs to handle 10,000 per seconds incoming records
> of average size of 3KB.
>
> -----------------------------------------------
> *Amol Suryawanshi*
> Java Developer
> amols@xxxxxxxxxxxxxxx
>
>
> *iProgrammer Solutions Pvt. Ltd.*
>
>
>
> *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society,
> Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - 411016,
> MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer*
> www.iprogrammer.com <sachin@xxxxxxxxxxxxxxx>
> ------------------------------------------------
>