OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cancel a Running dag


That won't stop them if they are already running in a celery worker or
already in your messaging queue backend (e.g. rabbitmq; redis), but it will
prevent the message to do them from being emitted again by the airflow
scheduler to your messaging queue backend. To be thorough you have to do
both - stop the scheduler from scheduling the tasks anymore (by failing
them individually and/or the DagRun in the metadata database) and, if you
want to make sure the tasks that already got picked up stop and don't try
again, you have to kill their worker processes and make sure your messaging
queue is clean of messages of that task type. If you don't care that any
already started or queued up tasks finish, you can simply doctor the
metadata database.

Laura

On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@xxxxxxxxx <ramandumcs@xxxxxxxxx
> wrote:

> Thanks Laura,
> We are using the CeleryExecutor. Just wondering if marking the
> TaskInstances as failed in metadata store would also work.
> -Raman
>
> On 2018/04/12 16:27:00, Laura Lorenz <llorenz@xxxxxxxxxxxxxxxx> wrote:
> > I use the CeleryExecutor and have used a mix of `celery control` and
> > messaging queue purges to kill the running tasks and prevent them from
> > being picked up by workers again (respectively), and doctor the DagRun to
> > failed to stop the scheduler from repopulating the message. I think if
> you
> > are using the Local or Sequential Executor you'd have to kill the
> scheduler
> > process.
> >
> > Laura
> >
> > On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <tedmiston@xxxxxxxxx>
> > wrote:
> >
> > > I don't think killing a currently running task is possible today.
> > >
> > > Of course you can pause it from the CLI or web UI so that future runs
> don't
> > > get triggered, but it sounds like that's not what you're looking for.
> > >
> > > Best,
> > > Taylor
> > >
> > > *Taylor Edmiston*
> > > Blog <http://blog.tedmiston.com> | Stack Overflow CV
> > > <https://stackoverflow.com/story/taylor> | LinkedIn
> > > <https://www.linkedin.com/in/tedmiston/> | AngelList
> > > <https://angel.co/taylor>
> > >
> > >
> > > On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@xxxxxxxxx <
> > > ramandumcs@xxxxxxxxx
> > > > wrote:
> > >
> > > > Hi All,
> > > > We have a use case to cancel the already running DAG. So is there any
> > > > recommended way to do so.
> > > >
> > > > Thanks,
> > > > Raman
> > > >
> > >
> >
>