osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re:Checkpointing Large State to S3



Hi Gregory,

could you share the TaskManager's log with us? It would be helpful to diagnost the problem. And which version are you using?

Best, Sihua
On 06/7/2018 06:42Gregory Fee<gfee@xxxxxxxx> wrote:
Hello Everyone!

I am running some streaming Flink jobs using SQL and the table api. I enabled incremental checkpointing to S3 via the RocksDBStateBackend. Even giving it an hour to checkpoint, the checkpoints all fail by timing out. Does anyone have an tips on how to configure the RocksDBStateBackend for best performance on S3? Or any tips with how to get checkpoints with large amounts of state to succeed?

Thanks!

--
Gregory Fee
Engineer
Lyft