[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: JVM metrics disappearing after job crash, restart

How are your metrics dimensionalized/named? Task managers often have UIDs generated for them. The task id dimension will change on restart. If you name your metric based on this 'task_id' there would be a discontinuity with the old metric.

On Wed, May 30, 2018 at 4:49 PM, Nikolas Davis <ndavis@xxxxxxxxxxxx> wrote:

We are seeing our task manager JVM metrics disappear over time. This last time we correlated it to our job crashing and restarting. I wasn't able to grab the failing exception to share. Any thoughts?

We track metrics through the MetricReporter interface. As far as I can tell this more or less only affects the JVM metrics. I.e. most / all other metrics continue reporting fine as the job is automatically restarted.

Nik Davis
Software Engineer
New Relic