Re: Subdag tasks unable to start
Thanks for replying : ) Thought I sent my messages to the wrong email or group.
> On May 8, 2018, at 4:41 PM, Arthur Wiedmer <arthur.wiedmer@xxxxxxxxx> wrote:
> Hi Victor,
> Usually, we expect the Schedules to match with Subdags, I am not sure
> Airflow supports heterogeneous schedules like this.
Good point. I don't understand the "* * * * *" schedule interval on the subdag either to be honest. We have used in other parts like that and I copied. I will match the parent DAG's schedule and see what it does.
> What you could have is one Triggering DAG, and a second DAG that it
> triggers at the right interval.
Thanks will also try this.
> Would you be able to explain further what you mean by simulate loops? Since
> Airflow tries to schedule Directed *Acyclic* Graphs, we don't really
> support a concept of loops
I know : ) So well, here's what I have (apologies for the bad diagram):
-> -> subdag-start
-> -> short-circuit-subdag (check if process should continue)
-> -> ... more tasks
-> -> triggerdaqgrun (with subdag id).
-> sqlsensor (check if process is still going)
In theory, it should work. In practice, it did not. The new subdag is added, but it never start. I have to start every task in it.