osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Wiring top-level DAGs together


Hello maintainers,

I need to have several identical (differing only in arguments) top-level
DAGs that can also be triggered together with following constraints /
assumptions

   - Individual top-level DAGs will have schedule_interval=None as they
   will only need occasional manual triggering
   - The series (wiring-up) of DAGs, however, needs to run daily
   - Order and number of DAGs in series is fixed (known ahead of writing
   code) and changes rarely (probably once in a month or so)

-------

What would be a good strategy to wire-up these top-level DAGs so that they
run can be run together as well as independently? (or is there something
fundamentally wrong with this idea?)

-------

I can think of SubDagOperator (problematic) and TriggerDagRunOperator
(satisfactory) but neither appear very tidy. Is there a better way? I am
open to complete re-structuring of my workflow as I am yet to deploy it.

------

Here's the link
<https://stackoverflow.com/questions/51325525/wiring-top-level-dags-together>
to my question on SO.
And here are some related links that I went through

   - Airflow, mark a task success or skip it before dag run
   <https://stackoverflow.com/questions/50934266/airflow-mark-a-task-success-or-skip-it-before-dag-run>
   - AIrflow - Splitting DAG definition across multiple files
   <https://stackoverflow.com/questions/41493605/airflow-splitting-dag-definition-across-multiple-files>
   - Airflow DAG success callback
   <https://stackoverflow.com/questions/48164160/airflow-dag-success-callback>
   - How to set dependencies between DAGs in Airflow?
   <https://stackoverflow.com/questions/38022323/how-to-set-dependencies-between-dags-in-airflow>
   - Airflow: When Your DAG is Far Behind The Schedule
   <https://medium.com/@hafizbadrie/airflow-when-your-dag-is-far-behind-the-schedule-ea11bf02e44c>


*Shubham Gupta*
Software Engineer
 zomato