osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Memory size for network buffers


Hi,

I'm trying to understand the intention behind the size parameters for
network buffers, specifically max, min and fraction. The reason we are
looking at it is an attempt to make the memory allocation elastic, so that
memory is allocated according to the actual number of buffers required
(within a range), without the need to tune this for every deployment.

As of Flink 1.5, there are 3 parameters, but they all result in a fixed
allocation, which is not what we were looking for.

Here is an example just to illustrate it:

taskmanager.network.memory.fraction: 0.000001
taskmanager.network.memory.min: 32768
taskmanager.network.memory.max: 1073741824
taskmanager.memory.segment-size: 8192

I wanted fraction to be out of the picture (but 0 isn't an acceptable
value).

Then set min to something tiny that my job will exceed and max to something
too large to reach. Unfortunately, that fails:

java.io.IOException: Insufficient number of network buffers: required 8,
but only 0 available. The total number of network buffers is currently set
to 4 of 8192 bytes each. You can increase this number by setting the
configuration keys 'taskmanager.network.memory.fraction',
'taskmanager.network.memory.min', and 'taskmanager.network.memory.max'.

So the question then is, why have min and max? Or is the intention to have
a different implementation in the future?

Thanks,
Thomas