osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: BigQuery streaming insert errors


Hi Carlos, 

Would an insert retry policy help you?
Please see this, https://beam.apache.org/documentation/sdks/javadoc/2.1.0/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicy.Context.html

Thanks, Gaurav

On Fri, Apr 6, 2018 at 8:13 AM, Pablo Estrada <pabloem@xxxxxxxxxx> wrote:
Im adding Cham as he might be knowledgeable about BQ IO, or he might be able to redirect to someone else.
Cham, do you have guidance for Carlos here?
Thanks
-P.


On Mon, Apr 2, 2018 at 11:08 AM Carlos Alonso <carlos@xxxxxxxxxxxxx> wrote:
And... where could I catch that exception?

Thanks!
On Mon, 2 Apr 2018 at 16:58, Ted Yu <yuzhihong@xxxxxxxxx> wrote:
Wouldn't the following code give you information about failed insertions (around line 790 in BigQueryServicesImpl) ?

      if (!allErrors.isEmpty()) {
        throw new IOException("Insert failed: " + allErrors);

Cheers

On Mon, Apr 2, 2018 at 7:16 AM, Carlos Alonso <carlos@xxxxxxxxxxxxx> wrote:
Hi everyone!!

I was wondering if there's any way to get the error why an insert (streaming) failed. Looking at the code I think there's currently no way to do that, as the BigQueryServicesImpl insertAll seems to discard the errors and just add the failed TableRow instances into the failedInserts list.

It would be very nice to have an "enriched" TableRow returned instead that contains the error information for further processing (in our use case we're saving the failed ones into a different table for further analysis)

Could this be added as an enhancement or similar Issue in GH/Jira? Any other ideas?

Thanks!

--
Got feedback? go/pabloem-feedback