[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Monitoring and Management Tools for Beam an Friends

Hi Benjamin
Please know there is an ongoing PR for a runner agnostic metrics feature

Le jeudi 02 mars 2017 à 16:26 +0000, Stas Levin a écrit :
Hi Benjamin,

This is somewhat of a hot topic lately, visibility FTW :)

My experience comes from doing Beam over Spark over Yarn, where we have been using Graphite + Grafana (4.0+ gives alerting as well). 

In terms of metrics, the Spark runner supports both Spark native metrics (default + Spark metric sources you configure, e.g. JVM), and Beam's, user-defined metrics, all of which are reported to whatever Spark sink(s) you have configured, in our case Graphite.

There were a few caveats we had to address, most of which had to do with associating an application's logical name and its application id assigned by Yarn. In other words, making all the metrics reported by a given application id, go under its logical name required some effort. However, once it's been done it allows for dashboards to be built around applications' logical names rather than around specific application ids which is very useful.

There are more caveats to speak of in the context of metrics and Beam over Spark (and streaming in particular), so let me know if you find the gory details interesting.


On Thu, Mar 2, 2017 at 5:37 PM Amit Sela <amitsela33@xxxxxxxxx> wrote:

On Thu, Mar 2, 2017 at 5:30 PM Jean-Baptiste Onofré <jb@xxxxxxxxxxxx> wrote:
Hi Benjamin,

It's a bit related to the Metric discussion on the dev@ mailing list.

Today, we leverage the monitoring and management provided by the
execution engine of the runner.

For instance, with the Spark runner, we can use the "regular" history
server, codehale reporter, etc. It's also possible to explicitly use a
sink/fn to send data to a backend (Graphite, elasticsearch, ...).

We are now discussing about a agnostic way of "pushing" metric to a
backend (something similar to what Karaf Decanter provides).


On 03/02/2017 04:13 PM, Stadin, Benjamin wrote:
> Hi,
> I’m trying to collect a list of open source monitoring and management
> tools for Beam and supported runners.
> What do you use in your daily routine, and what’s your experience?
> Regards
> Ben

Jean-Baptiste Onofré
Talend - http://www.talend.com