OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours


Not sure if this will help, but we had same issue 3 hours apart.
Problem was the DBMS (Postgres) was different timezone than
webserver/scheduler/workers.
When a new DAGRun was inserted, it was allowing DB to use current time for
execution date.

On Mon, Sep 10, 2018 at 2:10 PM, Bolke de Bruin <bdbruin@xxxxxxxxx> wrote:

> I cannot connect this to timezone issues yet. The trigger dagrun operator
> seems to work fine (checked the examples).
>
> B.
>
> > On 10 Sep 2018, at 08:19, Deng Xiaodong <xd.deng.r@xxxxxxxxx> wrote:
> >
> > Hi Goutam,
> >
> > Seems it’s due to the timezone setting?
> >
> > If that’s the case, you may try either adjusting your timezone setting or
> > “manually” adjusting using execution_date argument in
> TriggerDagRunOperator.
> >
> >
> > XD
> >
> > On Mon, Sep 10, 2018 at 14:00 Goutam Kumar Sahoo <
> > goutamkumar.sahoo@xxxxxxxxxxx> wrote:
> >
> >> HI All,
> >>
> >>
> >>
> >> Could anyone please address this issue ? We are really stuck here and
> not
> >> able to proceed further
> >>
> >>
> >>
> >> Thanks & Regards
> >>
> >> Goutam Sahoo
> >>
> >>
> >>
> >> *From:* Goutam Kumar Sahoo
> >> *Sent:* Friday, September 07, 2018 4:51 PM
> >> *To:* 'dev@xxxxxxxxxxxxxxxxxxxxxxxxxxxx' <dev@airflow.incubator.apache.
> org>;
> >> 'dev@xxxxxxxxxxxxxxxxxx' <dev@xxxxxxxxxxxxxxxxxx>
> >> *Subject:* TriggerDagRunOperator sub tasks are scheduled to run after
> few
> >> hours
> >>
> >>
> >>
> >> HI Experts
> >>
> >>
> >>
> >> In our project, we are trying to replicate the existing  job scheduling
> >> implemented in Microsoft Orchestrater to new scheduler called “Apache
> >> Airflow” .
> >>
> >>
> >>
> >> During this replication process we have an requirement to create Master
> >> DAG which in turn will call Other child DAGs based on different
> condition.
> >> We referred to the existing example (example_trigger_controller_dag,
> >> example_trigger_target_dag) available in Airflow and tried running them
> >> manually.
> >>
> >>
> >>
> >> When we triggered the controller DAG manually , it is got triggered
> >> immediately with execution time created in local timezone (PDT) whereas
> the
> >> child DAG is scheduled to run few hours later. Even though child DAGRUN
> is
> >> showing as running, it is doing nothing and waiting for the time to
> >> satisfy.  Our requirement is to trigger the Target/Child DAG run as
> soon it
> >> is triggered by controller DAG
> >>
> >>
> >>
> >>
> >>
> >> If you see here , the trigger time and execution time is having a 7
> hours
> >> GAP.
> >>
> >>
> >>
> >> We have tried to solve this many different ways but couldn’t succeed. We
> >> have asked for help in many forums but didn’t get any satisfactory
> answer
> >> form any of them
> >>
> >>
> >>
> >>
> >>
> >> Please help us ASAP as this issue is one of the show stopper for us .
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>
>