OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Using environment connection URL in operators


Deng,


I have only one node at present. 




Regards,

Abhishek 

> On 08-Aug-2018, at 5:30 AM, Deng Xiaodong <xd.deng.r@xxxxxxxxx> wrote:
> 
> Hi Abhishek,
> 
> Workers read connection info from its local environment variables. So if
> you have multiple Celery executor nodes (in distributed fashion), possibly
> check whether the same environement variable is exported on all of them?
> This may explain why the behaviour is random in your environment.
> 
> Folks please correct me if I’m wrong. Thanks.
> 
> 
> XD
> 
> On Wed, Aug 8, 2018 at 01:04 Abhishek Sinha <abhishek@xxxxxxxxxxxx> wrote:
> 
>> I am trying to use: MySqlToHiveTransfer Operator.
>> 
>> 
>> From the base_hook code, I see that there is a way to pass connection URL
>> via environment variables. The variable needs to be pre-fixed with:
>> AIRFLOW_CONN_ followed by the connection ID.
>> 
>> I have tried exporting this variable but the behaviour seems to be random.
>> In some instances of the DAG Run, the variable is present in os.environ but
>> in some of the runs it is not. Is this an issue?
>> 
>> We are on version 1.8.2 with Celery Executor.
>> 
>> 
>> 
>> 
>> Regards,
>> 
>> Abhishek
>> 
>>