osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Discussion] Move cluster creation functions from flink-cli.sh to other scripts


How would that look in detail given that I am a user who wants to start a
job cluster?

On Tue, Aug 21, 2018 at 12:09 PM Renjie Liu <liurenjie2008@xxxxxxxxx> wrote:

> But it's also dangerous if user need to submit job via the same cluster
> configuration. I mean we can use another script for starting job cluster,
> so that cluster administrators can separate environments.
> On Tue, Aug 21, 2018 at 5:53 PM Till Rohrmann <trohrmann@xxxxxxxxxx>
> wrote:
>
> > Hi Renjie,
> >
> > do you mean to not support deploying Flink clusters via
> > `FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job
> > cluster for a given job that way is quite useful.
> >
> > Cheers,
> > Till
> >
> > On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <liurenjie2008@xxxxxxxxx>
> wrote:
> >
> > > Hi:
> > > Flink 1.5.0 brings new deployment mode and we are able to provid a
> single
> > > cluster for our jobs which manages resources elastically. However, this
> > > also brings challenges for our cluster management since users can
> submit
> > > jobs or create yarn cluster in the same flink-cli.sh.
> > > Here is the motivation case for this issue:
> > >
> > > 1. Admins deployed a mesos session cluster using zookeeper ha mode
> > > 2. End user can submit jobs to this cluster
> > > 3. User can also create yarn cluster using the same conf mistakenly,
> > which
> > > brings our mesos session cluster down
> > >
> > > I propose to separate cluster manager functionalities from
> flink-cli.sh,
> > > and put them into other scripts.
> > > --
> > > Liu, Renjie
> > > Software Engineer, MVAD
> > >
> >
> --
> Liu, Renjie
> Software Engineer, MVAD
>