osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

BigQueryIO Failed Insert


Hey all,

I have been running into an issue with streaming writes to Big Query using a custom destination and I can't seem to find an appropriate solution in the docs so I was hoping someone here could help.

The Issue: Specifying a table destination which does not exist causes the message to fail to be 
                  inserted and there is no way to handle the error.

The problem is that the message does not appear in the `getFailedInserts()` function so I don't see a way of getting a handle on this error so that I could, for example, direct the message to a poison or deadletter queue.

Some code for reference....




Is there a way of handling this use case in the current beam SDK?

Thanks,
Strahinja Vladetic