osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: BigQueryIO Failed Insert


Hi Strahinja,

Looking at the code, not found is considered a transient error which will be retried until the table is created. Try using neverRetry and see if that gives you the results you expect?

Andrew

On 2018/05/04 17:16:39, Strahinja Vladetic <strahinja.vladetic@xxxxxxxxx> wrote: 
> Hey all,
> 
> I have been running into an issue with streaming writes to Big Query using
> a custom destination and I can't seem to find an appropriate solution in
> the docs so I was hoping someone here could help.
> 
> The Issue: Specifying a table destination which does not exist causes the
> message to fail to be
>                   inserted and there is no way to handle the error.
> 
> The problem is that the message does not appear in the `getFailedInserts()`
> function so I don't see a way of getting a handle on this error so that I
> could, for example, direct the message to a poison or deadletter queue.
> 
> Some code for reference....
> 
> 
> 
> 
> Is there a way of handling this use case in the current beam SDK?
> 
> Thanks,
> Strahinja Vladetic
>