OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Can a DAG be conditionally hidden from the UI?


@*jmeickle *can you share your code or any sample?  How are you returning
nothing?

I may have spoken too soon.  I can't seem to set dag to None, it fails with

*>>>>Tried to create relationships between tasks that don't have DAGs yet*


On Tue, Oct 9, 2018 at 12:57 PM Shah Altaf <mendhak@xxxxxxxxx> wrote:

>
>
> Thanks all, following your advice I've decided to do this right now with a
> decorator.  It looks crude so any suggestions are welcome.
>
> The decorator function checks an environment variable that we set on our
> servers and if it matches any of the passed in environment names, it'll
> allow the DAG to be created.
>
>
> def environments(env_names):
>     def restrict_environment(func):
>         def func_wrapper(*args, **kwargs):
> *            if( os.getenv("environment") in env_names):*
>                 return DAG(*args, **kwargs)
>         return func_wrapper
>     return restrict_environment
>
> *@environments(["env1","env2","production"])*
> def get_DAG(*args, **kwargs):
>     return None
>
> dag = get_DAG('my_dag', default_args=default_args, schedule_interval="30 5
> * * *")
>
>
> Regards
> Shah
>
>
> On Mon, Oct 8, 2018 at 5:19 PM William Pursell <williamp@xxxxxxxxx.invalid>
> wrote:
>
>> This would be a very desirable feature.  It's not just an issue of
>> differing environments, but also changing requirements.  A dag may be
>> used for a time, and then removed from the environment.  But if it's
>> still in the airflow db, it will get a row on the UI with a black "I'
>> indicating that the dag is missing.  As far as I know, then only way
>> to remove it is to manually edit the database.
>> On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <chris@xxxxxxxxxxxx> wrote:
>> >
>> > I like James solution better, but the initial thought I had was to
>> deploy
>> > airflowignore files to the environments to filter out files that should
>> not
>> > be processed when filling the DagBag.
>> >
>> > Chris
>> >
>> > On Mon, Oct 8, 2018 at 10:22 AM James Meickle
>> > <jmeickle@xxxxxxxxxxxxxx.invalid> wrote:
>> >
>> > > As long as the Airflow process can't find the DAG as a top-level
>> object in
>> > > the module, it won't be registered. For example, we have a function
>> that
>> > > returns DAGs; the function returns nothing if it's not in the right
>> > > environment.
>> > >
>> > > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <mendhak@xxxxxxxxx> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > tl;dr - Is it possible to conditionally hide a DAG from the UI
>> based on
>> > > an
>> > > > environment variable?
>> > > >
>> > > >
>> > > >
>> > > > Our team has a single repo with several DAGs in it and we deploy it
>> > > across
>> > > > multiple 'environments' (think dev, test, and other integration
>> > > > environments).  While most DAGs are meant to run everywhere, we do
>> have
>> > > > several which are meant to run in one and only one environment.  Of
>> > > course
>> > > > they are all paused, but it would be nice to declutter a bit for
>> > > ourselves.
>> > > >
>> > > > My question then - is it possible to conditionally hide a DAG from
>> the UI
>> > > > based on an environment variable or some flag somewhere.
>> > > >
>> > > > This is just wishful thinking - the dev could do something like
>> > > >
>> > > > dag = get_dag(...),and get_dag() would have a decorator like
>> > > > @only_run_in("integration4,dev,local")
>> > > >
>> > > > And that decorator returns some kind of None object or special DAG
>> which
>> > > > just doesn't appear in the list.
>> > > >
>> > > > Or perhaps some other way to accomplish this same effect - any ideas
>> > > would
>> > > > be appreciated.
>> > > >
>> > > > Thanks
>> > > >
>> > >
>>
>>