[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Airflow - YARN as an executor?

Happy to have it as a contrib executor. However, I personally think yarn is a dead end. It has a lot of catching up to do and all the momentum is with kubernetes. 


Verstuurd vanaf mijn iPad

> Op 24 apr. 2018 om 22:13 heeft Ruslan Dautkhanov <dautkhanov@xxxxxxxxx> het volgende geschreven:
> With Hadoop 3's Docker on YARN support, I think YARN becomes
> somewhat a competitor for Kubernetes.
> Great job on adding k8s support to Airflow.
> Very similarly I see Airflow could integrate with YARN and use
> its infrastructure as an "executor" .. have anyone explored feasibility of
> this approach?
> Thanks!
> Ruslan Dautkhanov