[jira] [Created] (FLINK-10981) Add or modify metrics to show the maximum usage of InputBufferPool/OutputBufferPool to help debugging back pressure
Yun Gao created FLINK-10981:
Summary: Add or modify metrics to show the maximum usage of InputBufferPool/OutputBufferPool to help debugging back pressure
Issue Type: Improvement
Components: Metrics, Network
Reporter: Yun Gao
Assignee: Yun Gao
Currently the network layer has provided two metrics items, namely _InputBufferPoolUsageGauge_ and _OutputBufferPoolUsageGauge_ to show the usage of input buffer pool and output buffer pool. When there are multiple inputs(SingleInputGate) or __ outputs(ResultPartition), the two metrics items show their average usage.
However, we found that the maximum usage of all the InputBufferPool or OutputBufferPool is also useful in debugging back pressure. Suppose we have a job with the following job graph:
A ---> B ----> C ---> D
Besides, also suppose D is very slow and thus cause back pressure, but E is very fast and F outputs few records, thus the usage of the corresponding input/output buffer pool is almost 0.
Then the average input/output buffer usage of each task will be:
A(100%) --> (100%) B (50%) --> (50%) C (100%) --> (100%) D
But the maximum input/output buffer usage of each task will be:
A(100%) --> (100%) B (100%) --> (100%) C (100%) --> (100%) D
Users will be able to find the slowest task by finding the first task whose input buffer usage is 100% but output usage is less than 100%.
If it is reasonable to show the maximum input/output buffer usage, I think there may be three options:
# Modify the current computation logic of _InputBufferPoolUsageGauge_ and _OutputBufferPoolUsageGauge._
# Add two __ new metrics items I_nputBufferPoolMaxUsageGauge and OutputBufferPoolUsageGauge._
# Try to show distinct usage for each input/output buffer pool.
and I think maybe the second option is the most preferred.
How do you think about that?
This message was sent by Atlassian JIRA