osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

SubdagOperator and Pools


Hi,

we have a SubdagOperator with lots of tasks in it. We want to limit the parallelism, with which these tasks execute. Therefore we created a pool and added the tasks within the SubdagOperator to this pool.

However, this setting is not respected (see image attached).

Now we am wondering why that is. In 'subdag_operator.py' on the master branch there is a comment that

    "Airflow pool is not honored by SubDagOperator."

This comment is not in the file in v1.9.0 (which I am using).

So this means that Pools are not respected for Subdags?

On the other handside it states that Subdags use the SequentialExecutor, which *should* execute tasks sequentially?

Can anyone clarify this, please?
And if pools do not work, what options do we have to limit parallelism in a Subdag?

Thanks in advance,
Andreas