[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

error handling flow in DAG

   I am coding various data flows and one of the requirements we have is to
have some error tasks happen when some of the tasks failure. These error
tasks are specific to the task that failed and are not a generic to the
whole DAG

   So for instance if I have a DAG that runs the following tasks

task_1 ----> task_2 ----> task_3

   If task_1 fails, then I want to run

task_1_failure_a ---> task_1_failure_b

   If task_2 fails, I do not need to do anything specific, but if task_3
fails, I need to run

task_3_failure_a ---> task_3_failure_b

   I already have a generic on_failure_callback task defined on all tasks
that handles alerting, but am stuck on the best way of handling a failure
flow for tasks

   The ways I have come up with of handling this are

Have a branch operator between each task with trigger_rule set to all_done.
The branch operator would then decide whether to go to next (success) task,
or to go down the failure branch

Put the failure tasks in a separate DAG with no schedule. Have a different
on_failure_callback for each task that would trigger the failure DAG for
that task and then do my generic error handling

   Does anybody have any thoughts on which of the above two approaches
would be best, or suggest an alternative way of doing this



Anthony Brown
Data Engineer BI Team - John Lewis
Tel : 0787 215 7305
This email is confidential and may contain copyright material of the John Lewis Partnership. 
If you are not the intended recipient, please notify us immediately and delete all copies of this message. 
(Please note that it is your responsibility to scan this message for viruses). Email to and from the
John Lewis Partnership is automatically monitored for operational and lawful business reasons.

John Lewis plc
Registered in England 233462
Registered office 171 Victoria Street London SW1E 5NN
Websites: https://www.johnlewis.com