IMO Beam would greatly benefit from having a monitoring UI. Individual runners (Spark, Dataflow) have their own UI's, which could be exposed through an API as you suggest.I also think there would be value in having a Beam-centric monitoring UI which focuses on concepts in the Beam model (PTransforms/PCollections, windows/triggers, bundles, metrics, state/timers, etc.) Dataflow's UI is pretty good, but I believe most users develop and debug locally (i.e. DirectRunner) before submitting to a cluster.On Mon, Oct 8, 2018 at 2:37 PM Winkelman, Kyle G <kyle.winkelman@xxxxxxxxx> wrote:
I am interested in exposing a monitoring url through the PipelineResult object. All of the documentation pages in the runners tab of beam.apache.org have a section dedicated to ‘Monitoring your job’ and most of those reference a web interface. If we were able to get a link to this web interface, we would be able to lead non-technical users of our application to a place where they can get some understanding of the progress of the job.
An example of this in the SparkPipelineResult would be returning the value of the javaSparkContext.sc().uiWebUrl().get(). I haven’t looked into other runners but if most of them have web interfaces it would make sense that they would be able to programmatically get a url to that web interface.
Looking for feedback on if this would be accepted and ways this can be accomplished in other runners.
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.--Got feedback? tinyurl.com/swegner-feedback