[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours


Dates are converted to UTC before submitting to the DB and are converted to UTC on retrieval. Is this on 1.10.0 release? Postgres will return datetimes with timezone information so conversion should go properly (MySQL won’t but we set it to UTC explicit)

B.

> On 10 Sep 2018, at 20:15, John Culver <john.culver@xxxxxxxxxxx> wrote:
> 
> Not sure if this will help, but we had same issue 3 hours apart.
> Problem was the DBMS (Postgres) was different timezone than
> webserver/scheduler/workers.
> When a new DAGRun was inserted, it was allowing DB to use current time for
> execution date.
> 
> On Mon, Sep 10, 2018 at 2:10 PM, Bolke de Bruin <bdbruin@xxxxxxxxx> wrote:
> 
>> I cannot connect this to timezone issues yet. The trigger dagrun operator
>> seems to work fine (checked the examples).
>> 
>> B.
>> 
>>> On 10 Sep 2018, at 08:19, Deng Xiaodong <xd.deng.r@xxxxxxxxx> wrote:
>>> 
>>> Hi Goutam,
>>> 
>>> Seems it’s due to the timezone setting?
>>> 
>>> If that’s the case, you may try either adjusting your timezone setting or
>>> “manually” adjusting using execution_date argument in
>> TriggerDagRunOperator.
>>> 
>>> 
>>> XD
>>> 
>>> On Mon, Sep 10, 2018 at 14:00 Goutam Kumar Sahoo <
>>> goutamkumar.sahoo@xxxxxxxxxxx> wrote:
>>> 
>>>> HI All,
>>>> 
>>>> 
>>>> 
>>>> Could anyone please address this issue ? We are really stuck here and
>> not
>>>> able to proceed further
>>>> 
>>>> 
>>>> 
>>>> Thanks & Regards
>>>> 
>>>> Goutam Sahoo
>>>> 
>>>> 
>>>> 
>>>> *From:* Goutam Kumar Sahoo
>>>> *Sent:* Friday, September 07, 2018 4:51 PM
>>>> *To:* 'dev@xxxxxxxxxxxxxxxxxxxxxxxxxxxx' <dev@airflow.incubator.apache.
>> org>;
>>>> 'dev@xxxxxxxxxxxxxxxxxx' <dev@xxxxxxxxxxxxxxxxxx>
>>>> *Subject:* TriggerDagRunOperator sub tasks are scheduled to run after
>> few
>>>> hours
>>>> 
>>>> 
>>>> 
>>>> HI Experts
>>>> 
>>>> 
>>>> 
>>>> In our project, we are trying to replicate the existing  job scheduling
>>>> implemented in Microsoft Orchestrater to new scheduler called “Apache
>>>> Airflow” .
>>>> 
>>>> 
>>>> 
>>>> During this replication process we have an requirement to create Master
>>>> DAG which in turn will call Other child DAGs based on different
>> condition.
>>>> We referred to the existing example (example_trigger_controller_dag,
>>>> example_trigger_target_dag) available in Airflow and tried running them
>>>> manually.
>>>> 
>>>> 
>>>> 
>>>> When we triggered the controller DAG manually , it is got triggered
>>>> immediately with execution time created in local timezone (PDT) whereas
>> the
>>>> child DAG is scheduled to run few hours later. Even though child DAGRUN
>> is
>>>> showing as running, it is doing nothing and waiting for the time to
>>>> satisfy.  Our requirement is to trigger the Target/Child DAG run as
>> soon it
>>>> is triggered by controller DAG
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> If you see here , the trigger time and execution time is having a 7
>> hours
>>>> GAP.
>>>> 
>>>> 
>>>> 
>>>> We have tried to solve this many different ways but couldn’t succeed. We
>>>> have asked for help in many forums but didn’t get any satisfactory
>> answer
>>>> form any of them
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Please help us ASAP as this issue is one of the show stopper for us .
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>>