osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Create Connections in Airflow operator at runtime


Hello maintainers

I have to run tasks on a remote EMR cluster, which is created (and later
destroyed) using EmrCreateJobFlowOperator
<https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/contrib/operators/emr_create_job_flow_operator.py>.
So the 'conn_id' in my SSHHook will be known only at runtime. Is it
possible to create (and also destroy) either of these from within an
Airflow operator?

   1. *Connection*
   <https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/models.py#L631>
(persisted
   in Airflow's db)
   2. *Environment Variable* (should be accessible to all downstream tasks
   and not just current task as told here
   <https://stackoverflow.com/questions/51312425/export-environment-variables-at-runtime-with-airflow#comment89615294_51312425>
   )


------------------------------

Here's the link
<https://stackoverflow.com/questions/53740885/create-and-use-connections-in-airflow-operator-at-runtime>
to my (elaborated) question on SO
Thank You

*Shubham Gupta*
Software Engineer
 zomato