OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Checkpointing Large State to S3


Hello Everyone!

I am running some streaming Flink jobs using SQL and the table api. I enabled incremental checkpointing to S3 via the RocksDBStateBackend. Even giving it an hour to checkpoint, the checkpoints all fail by timing out. Does anyone have an tips on how to configure the RocksDBStateBackend for best performance on S3? Or any tips with how to get checkpoints with large amounts of state to succeed?

Thanks!

--
Gregory Fee
Engineer
Lyft