I am coming across a use case where I may have to run more than100 parallel jobs(which may have different processing needs) on a Flink cluster.
My flink cluster, currently, has 1 Job Manager and 4/5 Task Managers depending on the processing needed is running on a Kubernetes cluster with 3 worker nodes.
I have few questions around this before I start looking at how I can execute this use case on such cluster configuration :
1) Is there any limitation w.r.t Job Manager while running 100 parallel jobs on a similar cluster?
2) Is there any limitation on the number of jobs that can be run on a cluster with 1 Job Manager?
3) Are there any recommendations around having hundreds of jobs on a single cluster?
4) Would I need multiple clusters for such a use case or are there any tuning parameters which can help for eg. Akka tuning parameters?
Thanks in advance,