[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Fwd: [DISCUSS] FLIP-6 Problems

---------- Forwarded message ---------
From: Renjie Liu <liurenjie2008@xxxxxxxxx>
Date: Tue, Jun 5, 2018 at 10:43 AM
Subject: [DISCUSS] FLIP-6 Problems
To: user <user@xxxxxxxxxxxxxxxx>


We've deployed flink 1.5.0 and tested the new cluster manager, it's really
great for flink to be elastic. However we've also found some problems that
blocks us from deploying it to production environment.

1. Task manager isolation. Currently flink allows different jobs to execute
on same task managers, this is unacceptable in production environment since
a faulty written job may kill task managers and affect other jobs.
2. Per job resource configuration. Currently flink session cluster can only
allocate same size and configuration task managers. This may waste a lot of
resources if we have a lot of jobs with different resource requirement.
3. Task manager's name is meanless.  This is a problem since we can't
monitor status of container in mesos environment.

One solution to the above problems is to use per job cluster, but a
centralized cluster manager can help to manage flink deployment and jobs

How you guys think about those? If the community agrees with us, we would
like to propose design and implementation to enhance the flink cluster
Liu, Renjie
Software Engineer, MVAD
Liu, Renjie
Software Engineer, MVAD