OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Flink memory management in table api


State is maintained in the configured state backend [1].

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/state_backends.html

2018-07-04 11:38 GMT+02:00 Amol S - iProgrammer <amols@xxxxxxxxxxxxxxx>:

> Hello fabian,
>
> Thanks for your quick response,
>
> According to above conversation flink will persist state forever for non
> windowed operations. I want to know how flink persiat the state i.e.
> Database or file system or in memory etc.
>
> On Wed, 4 Jul 2018 at 2:12 PM, Fabian Hueske <fhueske@xxxxxxxxx> wrote:
>
>> Hi Amol,
>>
>> The memory consumption depends on the query/operation that you are doing.
>> Time-based operations like group-window-aggregations,
>> over-window-aggregations, or window-joins can automatically clean up their
>> state once data is not no longer needed.
>> Operations such as non-windowed aggregations or joins have to persist all
>> data forever in state to guarantee absolute correctness.
>> However, you can also configure an idle state retention time [1] to
>> remove state that has not been accessed for a certain time.
>>
>> Regarding you questions:
>>
>> 1) Unless you configure the idle state retention time, state is kept as
>> long as needed to guarantee correctness, potentially forever.
>> 2) Queries use Flink's regular state features, i.e., you can configure
>> the RocksDBStateBackend to manage state on disk.
>> 3) This depends on your query and the distribution of your data.
>>
>> Best, Fabian
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-
>> release-1.5/dev/table/streaming.html#idle-state-retention-time
>>
>> 2018-07-04 7:46 GMT+02:00 Amol S - iProgrammer <amols@xxxxxxxxxxxxxxx>:
>>
>>> Hello folks,
>>>
>>> I am using flink table api to join multiple tables and create a single
>>> table from them. I have some doubts in my mind.
>>>
>>> 1. How long the query will maintain partial results per key and how it
>>> maintains state of each key?
>>>
>>> 2. If it is maintains state in memory then the memory will continuously
>>> grows and it leads to memory over head.
>>>
>>> 3. How much RAM server  needs to handle 10,000 per seconds incoming
>>> records
>>> of average size of 3KB.
>>>
>>> -----------------------------------------------
>>> *Amol Suryawanshi*
>>> Java Developer
>>> amols@xxxxxxxxxxxxxxx
>>>
>>>
>>> *iProgrammer Solutions Pvt. Ltd.*
>>>
>>>
>>>
>>> *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society,
>>> Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune -
>>> 411016,
>>> MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer*
>>> www.iprogrammer.com <sachin@xxxxxxxxxxxxxxx>
>>> ------------------------------------------------
>>>
>> --
>
> -----------------------------------------------
> *Amol Suryawanshi*
> Java Developer
> amols@xxxxxxxxxxxxxxx
>
>
> *iProgrammer Solutions Pvt. Ltd.*
>
>
>
> *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society,
> Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - 411016,
> MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer*
> www.iprogrammer.com <sachin@xxxxxxxxxxxxxxx>
> ------------------------------------------------
>