[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Benchmarking of Airflow Scheduler with Celery Executor


Hi Raman -

First, we’d be happy to help you test this out with Airflow. Or you could
do it yourself by using http://open.astronomer.io/airflow/ (w/ Docker
Engine + Docker Compose) to quickly spin up a test environment. Everything
is hooked to Prometheus/Grafana to monitor how the system reacts to your
workload.

-Ry
CEO, Astronomer

On April 12, 2018 at 12:23:46 PM, ramandumcs@xxxxxxxxx (ramandumcs@xxxxxxxxx)
wrote:

Hi All,
We have requirement to run 10k(s) of concurrent tasks. We are exploring
Airflow's Celery Executor for same. Horizontally Scaling of worker nodes
seem possible but it can only have one active scheduler.
So will Airflow scheduler be able to handle these many concurrent tasks.
Is there any benchmarking number around airflow scheduler's scalability.
Thanks,
Raman