osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cancel a Running dag


This is now fixed in master. Clearing tasks will now properly terminate a running task. If you pause the dag run no new tasks will be scheduled.

B.



Verstuurd vanaf mijn iPad

> Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <llorenz@xxxxxxxxxxxxxxxx> het volgende geschreven:
> 
> That won't stop them if they are already running in a celery worker or
> already in your messaging queue backend (e.g. rabbitmq; redis), but it will
> prevent the message to do them from being emitted again by the airflow
> scheduler to your messaging queue backend. To be thorough you have to do
> both - stop the scheduler from scheduling the tasks anymore (by failing
> them individually and/or the DagRun in the metadata database) and, if you
> want to make sure the tasks that already got picked up stop and don't try
> again, you have to kill their worker processes and make sure your messaging
> queue is clean of messages of that task type. If you don't care that any
> already started or queued up tasks finish, you can simply doctor the
> metadata database.
> 
> Laura
> 
> On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@xxxxxxxxx <ramandumcs@xxxxxxxxx
>> wrote:
> 
>> Thanks Laura,
>> We are using the CeleryExecutor. Just wondering if marking the
>> TaskInstances as failed in metadata store would also work.
>> -Raman
>> 
>>> On 2018/04/12 16:27:00, Laura Lorenz <llorenz@xxxxxxxxxxxxxxxx> wrote:
>>> I use the CeleryExecutor and have used a mix of `celery control` and
>>> messaging queue purges to kill the running tasks and prevent them from
>>> being picked up by workers again (respectively), and doctor the DagRun to
>>> failed to stop the scheduler from repopulating the message. I think if
>> you
>>> are using the Local or Sequential Executor you'd have to kill the
>> scheduler
>>> process.
>>> 
>>> Laura
>>> 
>>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <tedmiston@xxxxxxxxx>
>>> wrote:
>>> 
>>>> I don't think killing a currently running task is possible today.
>>>> 
>>>> Of course you can pause it from the CLI or web UI so that future runs
>> don't
>>>> get triggered, but it sounds like that's not what you're looking for.
>>>> 
>>>> Best,
>>>> Taylor
>>>> 
>>>> *Taylor Edmiston*
>>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV
>>>> <https://stackoverflow.com/story/taylor> | LinkedIn
>>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
>>>> <https://angel.co/taylor>
>>>> 
>>>> 
>>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@xxxxxxxxx <
>>>> ramandumcs@xxxxxxxxx
>>>>> wrote:
>>>> 
>>>>> Hi All,
>>>>> We have a use case to cancel the already running DAG. So is there any
>>>>> recommended way to do so.
>>>>> 
>>>>> Thanks,
>>>>> Raman
>>>>> 
>>>> 
>>> 
>>