[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Single Airflow Instance Vs Multiple Airflow Instance

So I would guess you have multiple team and could you give me an idea of
how many DAGs you actually have and how many runs per day ?

On Wed, Jun 6, 2018 at 11:30 AM, Arturo Michel <Arturo.Michel@xxxxxxxxxxxxxx
> wrote:

> We incorportated multi-tentacy in how we deploy dags.
> With version 0.9 you can have the following structure:
>  Project1/
>   - Dag_1
>   - Lib_for_dag_1
>  Project2/
>   - Dag_1
>   - lib_for_dag_1
> So that the team owning Project1 can independently and without inferring
> with others deploy (Project2) their own dags. If you incorporate this
> structure in your deployment process you could do this.
> It works fine.
> -----Original Message-----
> From: gauthiermartin86@ <gmail.com gauthiermartin86@xxxxxxxxx>
> Sent: Wednesday, June 6, 2018 4:24 PM
> To: dev@xxxxxxxxxxxxxxxxxx
> Subject: Single Airflow Instance Vs Multiple Airflow Instance
> Hi Everyone,
> We have been experimenting with airflow for about 6 months now.
> We are planning to have multiple departments to use it. Since we don't
> have any internal experience with Airflow we are wondering if single
> instance per department is more suited than single instance with
> multi-tenancy? We have been aware about the upcoming release of airflow
> 1.10 and changes that will be made to the RBAC which will be more suited
> for multi-tenancy.
> Any advice on this ? Any tips could be helpful to us.
> This e-mail message and any attachments are confidential and are for the
> exclusive use of the addressee only.  If you are not the intended
> recipient, you should not use the content, place any reliance on it or
> disclose it to anyone else.  Please notify the sender immediately by
> replying to it and then ensure that it is deleted from your system
> (including any attachments).

Martin Gauthier

C: 438-499-1749