osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Discussion] Move cluster creation functions from flink-cli.sh to other scripts


Hi Renjie,

do you mean to not support deploying Flink clusters via
`FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job
cluster for a given job that way is quite useful.

Cheers,
Till

On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <liurenjie2008@xxxxxxxxx> wrote:

> Hi:
> Flink 1.5.0 brings new deployment mode and we are able to provid a single
> cluster for our jobs which manages resources elastically. However, this
> also brings challenges for our cluster management since users can submit
> jobs or create yarn cluster in the same flink-cli.sh.
> Here is the motivation case for this issue:
>
> 1. Admins deployed a mesos session cluster using zookeeper ha mode
> 2. End user can submit jobs to this cluster
> 3. User can also create yarn cluster using the same conf mistakenly, which
> brings our mesos session cluster down
>
> I propose to separate cluster manager functionalities from flink-cli.sh,
> and put them into other scripts.
> --
> Liu, Renjie
> Software Engineer, MVAD
>