OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Can a DAG be conditionally hidden from the UI?


This would be a very desirable feature.  It's not just an issue of
differing environments, but also changing requirements.  A dag may be
used for a time, and then removed from the environment.  But if it's
still in the airflow db, it will get a row on the UI with a black "I'
indicating that the dag is missing.  As far as I know, then only way
to remove it is to manually edit the database.
On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <chris@xxxxxxxxxxxx> wrote:
>
> I like James solution better, but the initial thought I had was to deploy
> airflowignore files to the environments to filter out files that should not
> be processed when filling the DagBag.
>
> Chris
>
> On Mon, Oct 8, 2018 at 10:22 AM James Meickle
> <jmeickle@xxxxxxxxxxxxxx.invalid> wrote:
>
> > As long as the Airflow process can't find the DAG as a top-level object in
> > the module, it won't be registered. For example, we have a function that
> > returns DAGs; the function returns nothing if it's not in the right
> > environment.
> >
> > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <mendhak@xxxxxxxxx> wrote:
> >
> > > Hi all,
> > >
> > > tl;dr - Is it possible to conditionally hide a DAG from the UI based on
> > an
> > > environment variable?
> > >
> > >
> > >
> > > Our team has a single repo with several DAGs in it and we deploy it
> > across
> > > multiple 'environments' (think dev, test, and other integration
> > > environments).  While most DAGs are meant to run everywhere, we do have
> > > several which are meant to run in one and only one environment.  Of
> > course
> > > they are all paused, but it would be nice to declutter a bit for
> > ourselves.
> > >
> > > My question then - is it possible to conditionally hide a DAG from the UI
> > > based on an environment variable or some flag somewhere.
> > >
> > > This is just wishful thinking - the dev could do something like
> > >
> > > dag = get_dag(...),and get_dag() would have a decorator like
> > > @only_run_in("integration4,dev,local")
> > >
> > > And that decorator returns some kind of None object or special DAG which
> > > just doesn't appear in the list.
> > >
> > > Or perhaps some other way to accomplish this same effect - any ideas
> > would
> > > be appreciated.
> > >
> > > Thanks
> > >
> >