[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Discussion] Move cluster creation functions from flink-cli.sh to other scripts

Flink 1.5.0 brings new deployment mode and we are able to provid a single
cluster for our jobs which manages resources elastically. However, this
also brings challenges for our cluster management since users can submit
jobs or create yarn cluster in the same flink-cli.sh.
Here is the motivation case for this issue:

1. Admins deployed a mesos session cluster using zookeeper ha mode
2. End user can submit jobs to this cluster
3. User can also create yarn cluster using the same conf mistakenly, which
brings our mesos session cluster down

I propose to separate cluster manager functionalities from flink-cli.sh,
and put them into other scripts.
Liu, Renjie
Software Engineer, MVAD