[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Accessing attempted metrics from within a DoFn

Adding Andrea as she may have thoughts on this.

On Wed, Aug 29, 2018 at 10:43 AM Robin Qiu <robinyq@xxxxxxxxxx> wrote:
Hi everyone,

I am writing a test [1] for the support of @RequiresStableInput annotation in Java SDK [2]. In the test I need to have a ParDo make some side effects and then fail. Currently, the only way to persist the side effect during retries without depending on external states is using attempted metrics. However, attempted metrics cannot be accessed from within a DoFn. So I am wondering if we should make an improvement on this by, for example, exposing a job handler to DoFn.

As Luke mentioned in another email thread, this improvement will also be useful since:
* cancelling a pipeline from within the pipeline is useful
* starting a new job against the existing runner from in a pipeline is useful
* accessing attempted metrics to test DoFn's with side effects is useful for error handling testing

What do you think? All suggestions are appreciated.