osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Hundreds of parallel jobs on Flink Cluster


Hi,

I am coming across a use case where I may have to run more than100 parallel jobs(which may have different processing needs) on a Flink cluster. 

My flink cluster, currently, has 1 Job Manager and 4/5 Task Managers depending on the processing needed is running on a Kubernetes cluster with 3 worker nodes.   

I have few questions around this before I start looking at how I can execute this use case on such cluster configuration : 

1) Is there any limitation w.r.t  Job Manager while running 100 parallel jobs on a similar cluster? 
2) Is there any limitation on the number of jobs that can be run on a cluster with 1 Job Manager? 
3) Are there any recommendations around having hundreds of jobs on a single cluster?
4) Would I need multiple clusters for such a use case or are there any tuning parameters which can help for eg. Akka tuning parameters? 

Thanks in advance,

Chirag