OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Submitting Flink application on YARN parameter


Hi,

You have to enable CPU scheduling in YARN, otherwise it always shows that only 1 CPU is allocated for each container,

regardless of how many Flink try to allocated.


TaskManager memory is 1400MB, but Flink reserves some amount for for off-heap memory, so the actual heap size is smaller.

This is controlled by 2 settings:

containerized.heap-cutoff-min: default 600MB
containerized.heap-cutoff-ratio: default 15% of TM's memory

That's why your TM's heap size is limitted to ~800MB (1400 - 600)


Regards,

Kien


On 4/29/2018 12:36 PM, Soheil Pourbafrani wrote:
Hi, I have an Flink .jar file and I submit it on yarn cluster using the command:

flink run -m yarn-cluster -yn 5 -yjm 768 -ytm 1400 -ys 2 -yqu streamQ my_program.jar

According to the submitting command I expect:

// It will create 5 containers ----> satisfied
// each container should use 2 core --> not satisfied according to the YARN webui but in jobmanager webui it shows       each taskmanager has 2 slot
// jobmanager should use 768m ram --> satisfied
// each taskmanager should use 1400m ram --> satisfied according to the YARN         webui but in jobmanager webui it shows for each taskmanager:  767m heap memory and 478m used by Flink


Why the number of used cores and memory is not as desired?