osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How to use RocksDBStateBackend predefined options


Hi all

We already found the programmatic way to configure RocksDB was not so friendly in our Alibaba's production environment, and refactored it by wrapping customer configurations for RocksDB. We pre-wrapped configurations such as block cache size, whether to cache index&filter into block cache size, number of background flush threads, and could get the them from TaskManagerConfiguration if users want to change the default values. But I think our community's way using options factory could also make sense if sophisticated user want to make fine-grained tuning which might use configuration not included in our pre-wrapped configuration. We might combine these two kind of operations together.

I'd like to create a JIRA about this improvement, what's your opinions?

Best
Yun Tang

________________________________
From: Thomas Weise <thw@xxxxxxxxxx>
Sent: Tuesday, November 13, 2018 1:53
To: dev@xxxxxxxxxxxxxxxx
Cc: uce@xxxxxxxxxx
Subject: Re: How to use RocksDBStateBackend predefined options

Sounds good. Perhaps it would also be good to allow the user to specify an
options factory in flink-conf.yaml for more flexibility?

Thomas

On Mon, Nov 12, 2018 at 9:48 AM Stefan Richter <s.richter@xxxxxxxxxxxxxxxxx>
wrote:

> Hi,
>
> Ufuk is right, for historical reasons there is currently only the
> programatic way but I think nothing speaks fundamentally against offering
> configuration via config in the future (maybe just a lot of config keys
> must be introduced to cover all options).
>
> Best,
> Stefan
>
> > On 9. Nov 2018, at 22:52, Ufuk Celebi <uce@xxxxxxxxxx> wrote:
> >
> > Hey Thomas,
> >
> > On Fri, Nov 9, 2018 at 6:07 PM Thomas Weise <thw@xxxxxxxxxx> wrote:
> >> Is there a way to activate the predefined options via configuration /
> flink-
> >> conf.yaml? Or only programmatically, like in [4]? The difficulty with
> the
> >> programmatic route (assuming this works now), is that in my case the
> client
> >> is Beam and I'm not writing the code that submits the job.
> >
> > AFAIK no. You can only do it programmatically at the moment [1].
> > Having the option to configure all settings through the configuration
> > file seems to be a valid feature request to me.
> >
> > @Stefan Richter (cc'd): Are there any reasons that speak against
> > exposing these options via the configuration file in your opinion?
> >
> > Best,
> >
> > Ufuk
> >
> > [1] Looking at the code in [2] and [3], the only options that are
> > exposed via the configuration file are local directories and the timer
> > service factory.
> > [2]
> https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBOptions.java
> > [3]
> https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java
>
>