osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Use KubernetesExecutor to launch tasks into a Dask cluster in Kubernetes


Hi Kyle,

So you have a static Dask cluster running your k8s cluster? Is there any
reason you wouldn't just launch the Dask cluster for the job you're running
and then tear it down? I feel like with k8s the elasticity is one of the
main benefits.

On Fri, Apr 27, 2018 at 12:32 PM Kyle Hamlin <hamlin.kn@xxxxxxxxx> wrote:

> Hi all,
>
> If I have a Kubernetes cluster running in DCOC and a Dask cluster running
> in that same Kubernetes cluster is it possible/does it makes sense to use
> the KubernetesExecutor to launch tasks into the Dask cluster (these are ML
> jobs with sklearn)? I feel like there is a bit of inception going on here
> in my mind and I just want to make sure a setup like this makes sense?
> Thanks in advance for anyone's input!
>