osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: About the project support in Airflow


We use a similar approach like Bolke mentioned with running multiple
Airflow instances.

I haven't read the Pandora article yet, but we have an Astronomer Open
Edition (fully open source) that bundles similar tools like Prometheus,
Grafana, Celery, etc with Airflow and a Docker Compose file if you're
looking to get a setup like that up and running quickly.

https://github.com/astronomerio/astronomer/blob/master/examples/airflow-enterprise/docker-compose.yml
https://github.com/astronomerio/astronomer

*Taylor Edmiston*
Blog <http://blog.tedmiston.com> | Stack Overflow CV
<https://stackoverflow.com/story/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor>


On Tue, Apr 24, 2018 at 3:30 PM, Maxime Beauchemin <
maximebeauchemin@xxxxxxxxx> wrote:

> Related blog post about multi-tenant Airflow deployment out of Pandora:
> https://engineering.pandora.com/apache-airflow-at-pandora-1d7a844d68ee
>
> On Tue, Apr 24, 2018 at 10:20 AM, Bolke de Bruin <bdbruin@xxxxxxxxx>
> wrote:
>
> > My suggestion would be to deploy airflow per project. You could even use
> > airflow to manage your ci/cd pipeline.
> >
> > B.
> >
> > Sent from my iPhone
> >
> > > On 24 Apr 2018, at 18:33, Maxime Beauchemin <
> maximebeauchemin@xxxxxxxxx>
> > wrote:
> > >
> > > People have been talking about namespacing DAGs in the past. I'd
> > recommend
> > > using tags (many to many) instead of categories/projects (one to many).
> > >
> > > It should be fairly easy to add this feature. One question is whether
> > tags
> > > are defined as code or in the UI/db only.
> > >
> > > Max
> > >
> > >> On Tue, Apr 24, 2018 at 1:48 AM, Song Liu <songliu@xxxxxxxxxxx>
> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Basically the DAGs are created for a project purpose, so if I have
> many
> > >> different projects, will the Airflow support the Project concept and
> > >> organize them separately ?
> > >>
> > >> Is this a known requirement or any plan for this already ?
> > >>
> > >> Thanks,
> > >> Song
> > >>
> >
>