osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cancel a Running dag


Like Bolke said, it has been fixed in master. One of the perquisites is
support by the operator. For example, the Spark operator has implemented
how to kill the Spark job on YARN, Local and Kubernetes. If you are running
something else, you might want to check if this is implemented.

Implemented on_kill: https://github.com/apache/incubator-airflow/pull/3204
An example of the on_kill:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L485-L534

Cheers, Fokko

2018-04-12 21:19 GMT+02:00 Bolke de Bruin <bdbruin@xxxxxxxxx>:

> This is now fixed in master. Clearing tasks will now properly terminate a
> running task. If you pause the dag run no new tasks will be scheduled.
>
> B.
>
>
>
> Verstuurd vanaf mijn iPad
>
> > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <llorenz@xxxxxxxxxxxxxxxx>
> het volgende geschreven:
> >
> > That won't stop them if they are already running in a celery worker or
> > already in your messaging queue backend (e.g. rabbitmq; redis), but it
> will
> > prevent the message to do them from being emitted again by the airflow
> > scheduler to your messaging queue backend. To be thorough you have to do
> > both - stop the scheduler from scheduling the tasks anymore (by failing
> > them individually and/or the DagRun in the metadata database) and, if you
> > want to make sure the tasks that already got picked up stop and don't try
> > again, you have to kill their worker processes and make sure your
> messaging
> > queue is clean of messages of that task type. If you don't care that any
> > already started or queued up tasks finish, you can simply doctor the
> > metadata database.
> >
> > Laura
> >
> > On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@xxxxxxxxx <
> ramandumcs@xxxxxxxxx
> >> wrote:
> >
> >> Thanks Laura,
> >> We are using the CeleryExecutor. Just wondering if marking the
> >> TaskInstances as failed in metadata store would also work.
> >> -Raman
> >>
> >>> On 2018/04/12 16:27:00, Laura Lorenz <llorenz@xxxxxxxxxxxxxxxx> wrote:
> >>> I use the CeleryExecutor and have used a mix of `celery control` and
> >>> messaging queue purges to kill the running tasks and prevent them from
> >>> being picked up by workers again (respectively), and doctor the DagRun
> to
> >>> failed to stop the scheduler from repopulating the message. I think if
> >> you
> >>> are using the Local or Sequential Executor you'd have to kill the
> >> scheduler
> >>> process.
> >>>
> >>> Laura
> >>>
> >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <tedmiston@xxxxxxxxx
> >
> >>> wrote:
> >>>
> >>>> I don't think killing a currently running task is possible today.
> >>>>
> >>>> Of course you can pause it from the CLI or web UI so that future runs
> >> don't
> >>>> get triggered, but it sounds like that's not what you're looking for.
> >>>>
> >>>> Best,
> >>>> Taylor
> >>>>
> >>>> *Taylor Edmiston*
> >>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV
> >>>> <https://stackoverflow.com/story/taylor> | LinkedIn
> >>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
> >>>> <https://angel.co/taylor>
> >>>>
> >>>>
> >>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@xxxxxxxxx <
> >>>> ramandumcs@xxxxxxxxx
> >>>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>> We have a use case to cancel the already running DAG. So is there any
> >>>>> recommended way to do so.
> >>>>>
> >>>>> Thanks,
> >>>>> Raman
> >>>>>
> >>>>
> >>>
> >>
>