osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

How to use RocksDBStateBackend predefined options


Hi,

We are observing unbounded memory consumption with the default
RocksDBStateBackend configuration [1]. According to [2] it is possible to
control that through custom options or by using the predefined options from
[3].

Is there a way to activate the predefined options via configuration / flink-
conf.yaml? Or only programmatically, like in [4]? The difficulty with the
programmatic route (assuming this works now), is that in my case the client
is Beam and I'm not writing the code that submits the job.

Thanks,
Thomas

[1] https://stackoverflow.com/questions/50812837/flink-taskmanager
-out-of-memory-and-memory-configuration/53229810#53229810

[2] https://ci.apache.org/projects/flink/flink
-docs-stable/ops/state/large_state_tuning.html#tuning-rocksdb

[3] https://github.com/apache/flink/blob/release-1.5/flink-state-backends/
flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib
/streaming/state/PredefinedOptions.java#L44

[4] https://stackoverflow.com/questions/44819521/how-to-set-
rocksdbstatebackend-parameter-in-flink