[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Accessing attempted metrics from within a DoFn


I asked myself the same thing, and indeed there is no way of accessing the metrics from within the pipeline itself. The only access you can have is directly to the MetricCell like that (1) but it is runner facing, it is not user facing, so not helping for your test.
So I agree with you: such a job handler would be useful.

[1] https://github.com/apache/beam/blob/6497b0b24888178441f05d79ab101dc40760388b/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataUtils.java#L95


Le mercredi 29 août 2018 à 10:43 -0700, Robin Qiu a écrit :
Hi everyone,

I am writing a test [1] for the support of @RequiresStableInput annotation in Java SDK [2]. In the test I need to have a ParDo make some side effects and then fail. Currently, the only way to persist the side effect during retries without depending on external states is using attempted metrics. However, attempted metrics cannot be accessed from within a DoFn. So I am wondering if we should make an improvement on this by, for example, exposing a job handler to DoFn.

As Luke mentioned in another email thread, this improvement will also be useful since:
* cancelling a pipeline from within the pipeline is useful
* starting a new job against the existing runner from in a pipeline is useful
* accessing attempted metrics to test DoFn's with side effects is useful for error handling testing

What do you think? All suggestions are appreciated.