OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Obey DAG concurrency during backfill


I've found this thread on SO that the DAG concurrency setting is not obeyed
from external triggers (like a backfill).
https://stackoverflow.com/questions/45752912/how-control-dag-concurrency-in-airflow

I'm increasing my "parallelism" and "dag_concurrency" as we increase the
number of DAGs running, expecting to limit simultaneous tasks per DAG via
the `concurrency` attribute. I'd like this enforced always though, not just
when triggered by the scheduler.

Other than assigning all tasks to pools (which I'll probably do soon) is
there another way to enforce the concurrency?

Thanks