osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cancel a Running dag


Thanks Bolke,
Will it become part of Airflow 1.10 release. Is there any tentative timeline for same.
-Raman Gupta

On 2018/04/12 19:19:07, Bolke de Bruin <bdbruin@xxxxxxxxx> wrote: 
> This is now fixed in master. Clearing tasks will now properly terminate a running task. If you pause the dag run no new tasks will be scheduled.
> 
> B.
> 
> 
> 
> Verstuurd vanaf mijn iPad
> 
> > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <llorenz@xxxxxxxxxxxxxxxx> het volgende geschreven:
> > 
> > That won't stop them if they are already running in a celery worker or
> > already in your messaging queue backend (e.g. rabbitmq; redis), but it will
> > prevent the message to do them from being emitted again by the airflow
> > scheduler to your messaging queue backend. To be thorough you have to do
> > both - stop the scheduler from scheduling the tasks anymore (by failing
> > them individually and/or the DagRun in the metadata database) and, if you
> > want to make sure the tasks that already got picked up stop and don't try
> > again, you have to kill their worker processes and make sure your messaging
> > queue is clean of messages of that task type. If you don't care that any
> > already started or queued up tasks finish, you can simply doctor the
> > metadata database.
> > 
> > Laura
> > 
> > On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@xxxxxxxxx <ramandumcs@xxxxxxxxx
> >> wrote:
> > 
> >> Thanks Laura,
> >> We are using the CeleryExecutor. Just wondering if marking the
> >> TaskInstances as failed in metadata store would also work.
> >> -Raman
> >> 
> >>> On 2018/04/12 16:27:00, Laura Lorenz <llorenz@xxxxxxxxxxxxxxxx> wrote:
> >>> I use the CeleryExecutor and have used a mix of `celery control` and
> >>> messaging queue purges to kill the running tasks and prevent them from
> >>> being picked up by workers again (respectively), and doctor the DagRun to
> >>> failed to stop the scheduler from repopulating the message. I think if
> >> you
> >>> are using the Local or Sequential Executor you'd have to kill the
> >> scheduler
> >>> process.
> >>> 
> >>> Laura
> >>> 
> >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <tedmiston@xxxxxxxxx>
> >>> wrote:
> >>> 
> >>>> I don't think killing a currently running task is possible today.
> >>>> 
> >>>> Of course you can pause it from the CLI or web UI so that future runs
> >> don't
> >>>> get triggered, but it sounds like that's not what you're looking for.
> >>>> 
> >>>> Best,
> >>>> Taylor
> >>>> 
> >>>> *Taylor Edmiston*
> >>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV
> >>>> <https://stackoverflow.com/story/taylor> | LinkedIn
> >>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
> >>>> <https://angel.co/taylor>
> >>>> 
> >>>> 
> >>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@xxxxxxxxx <
> >>>> ramandumcs@xxxxxxxxx
> >>>>> wrote:
> >>>> 
> >>>>> Hi All,
> >>>>> We have a use case to cancel the already running DAG. So is there any
> >>>>> recommended way to do so.
> >>>>> 
> >>>>> Thanks,
> >>>>> Raman
> >>>>> 
> >>>> 
> >>> 
> >> 
>