osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Discussion] Move cluster creation functions from flink-cli.sh to other scripts


But it's also dangerous if user need to submit job via the same cluster
configuration. I mean we can use another script for starting job cluster,
so that cluster administrators can separate environments.
On Tue, Aug 21, 2018 at 5:53 PM Till Rohrmann <trohrmann@xxxxxxxxxx> wrote:

> Hi Renjie,
>
> do you mean to not support deploying Flink clusters via
> `FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job
> cluster for a given job that way is quite useful.
>
> Cheers,
> Till
>
> On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <liurenjie2008@xxxxxxxxx> wrote:
>
> > Hi:
> > Flink 1.5.0 brings new deployment mode and we are able to provid a single
> > cluster for our jobs which manages resources elastically. However, this
> > also brings challenges for our cluster management since users can submit
> > jobs or create yarn cluster in the same flink-cli.sh.
> > Here is the motivation case for this issue:
> >
> > 1. Admins deployed a mesos session cluster using zookeeper ha mode
> > 2. End user can submit jobs to this cluster
> > 3. User can also create yarn cluster using the same conf mistakenly,
> which
> > brings our mesos session cluster down
> >
> > I propose to separate cluster manager functionalities from flink-cli.sh,
> > and put them into other scripts.
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
> >
>
-- 
Liu, Renjie
Software Engineer, MVAD