OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Subdag tasks unable to start


Thanks for replying : ) Thought I sent my messages to the wrong email or group.

> On May 8, 2018, at 4:41 PM, Arthur Wiedmer <arthur.wiedmer@xxxxxxxxx> wrote:
> 
> Hi Victor,
> 
> Usually, we expect the Schedules to match with Subdags, I am not sure
> Airflow supports heterogeneous schedules like this.

Good point. I don't understand the "* * * * *" schedule interval on the subdag either to be honest. We have used in other parts like that and I copied. I will match the parent DAG's schedule and see what it does.

> What you could have is one Triggering DAG, and a second DAG that it
> triggers at the right interval.

Thanks will also try this.

> Would you be able to explain further what you mean by simulate loops? Since
> Airflow tries to schedule Directed *Acyclic* Graphs, we don't really
> support a concept of loops

I know : ) So well, here's what I have (apologies for the bad diagram):

dag
-> start-dag
-> subdag
-> -> subdag-start
-> -> short-circuit-subdag (check if process should continue)
-> -> ... more tasks
-> -> triggerdaqgrun (with subdag id).
-> sqlsensor (check if process is still going)

In theory, it should work. In practice, it did not. The new subdag is added, but it never start. I have to start every task in it.

> Best,
> Arthur