1. It would be beneficial I think to drop a line somewhere in the
docs (probably on the production ready checklist as well as the HA
page) explaining that enabling zookeeper "highavailability" allows
for your jobs to restart automatically after a jobmanager crash or
restart. We had spent some cycles trying to implement job
restarting and watchdogs (poorly) when I discoverd this from a
flink forward presentation on youtube.
2. I seem to have found some odd behavior with HA and then found
something that works, but I can't explain why. The clifnotes
version is that I took an existing standalone cluster with a
single JM and modified with high availability zookeeper mode. The
same flink-conf.yaml file is used on all nodes (including JM).
This seemed to work fine, I restarted the JM (jm0) and the jobs
relaunched when it came back. Easy! Then I deployed a second JM
(jm1). Once I modified `masters`, set the HA rpc port range and
opened those ports on the firewall for both jobmanagers, but left
`jobmanager.rpc.address` the original value, `jm0` on all nodes.
I then observed that jm0 worked fine, taskmanagers connected to it
and jobs ran. jm1 did not 301 me to jm0 however, it displayed a
dashboard (no jobs, no tm). When I stopped jm0, the jobs show up
on jm1 as RESTARTING, but the taskmanagers never attach to jm1.
In the logs, all nodes, including jm1, had messages about trying
to reach jm0. From the documentation and various comments I've
seen, `jobmanager.rpc.address` should be ignored. However,
commenting it out entirely lead to jobmanagers crashing at boot,
setting to `localhost` caused all the taskmanagers to log messages
about trying to connect to the jobmanager at localhost. What
finally worked was to set the value to the hostname where the
flink-conf.yaml was individually, even on the taskmanagers.
Does this seem like a bug?
Just a hunch, but is there something called an "akka leader" that is different from the jobmanager leader, and could it be somehow defaulting its value over to jobmanager.rpc.address?