[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Monitoring Url


I think it's something I proposed while ago. Maybe some of you remember the discussion and use of Karaf Decanter like.

The idea was to collect (collector) metrics and data and dispatch to appender. The UI could be on the appender side.

With kind of Karaf Decanter approach we can have UI (leveraging other tools) covering monitoring, bam, auditing, etc.

Just my €0.01 😄😄

Le 9 oct. 2018, à 22:34, Scott Wegner <scott@xxxxxxxxxx> a écrit:
IMO Beam would greatly benefit from having a monitoring UI. Individual runners (Spark, Dataflow) have their own UI's, which could be exposed through an API as you suggest. 

I also think there would be value in having a Beam-centric monitoring UI which focuses on concepts in the Beam model (PTransforms/PCollections, windows/triggers, bundles, metrics, state/timers, etc.) Dataflow's UI is pretty good, but I believe most users develop and debug locally (i.e. DirectRunner) before submitting to a cluster.

On Mon, Oct 8, 2018 at 2:37 PM Winkelman, Kyle G <kyle.winkelman@xxxxxxxxx> wrote:



I am interested in exposing a monitoring url through the PipelineResult object. All of the documentation pages in the runners tab of beam.apache.org have a section dedicated to ‘Monitoring your job’ and most of those reference a web interface. If we were able to get a link to this web interface, we would be able to lead non-technical users of our application to a place where they can get some understanding of the progress of the job.


An example of this in the SparkPipelineResult would be returning the value of the javaSparkContext.sc().uiWebUrl().get(). I haven’t looked into other runners but if most of them have web interfaces it would make sense that they would be able to programmatically get a url to that web interface.


Looking for feedback on if this would be accepted and ways this can be accomplished in other runners.



Kyle Winkelman


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.