OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Suggesting change on BQ streaming dead-letter


I agree .- one reason why the current InsertRetryPolicy takes a Context object (InsertRetryPolicy.java) is exactly so we can augment that information with extra information, such as the current retry count. One tricky issue though is that we can only accurately know the retry count in the current bundle. If the entire bundle fails (because an exception is thrown or a worker crashes), then the retry count will probably start from 0 again.

Reuven

On Mon, May 14, 2018 at 1:27 AM Carlos Alonso <carlos@xxxxxxxxxxxxx> wrote:
Hi devs!!

After working on this PR https://github.com/apache/beam/pull/5341 I've realised that the dead-letter functionality on BQ streaming could be improved a little bit with by retrying a configurable amount of times and then, if still failing, return the error via getFailedInserts rather than throwing an exception as currently happens...

I'd be happy to work on it but before kicking off I'd like to have some of your feedback...

Thanks!