osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Multiple hosts for a single connection


Hi Ben,

If you would like to set multiple connections with the same *conn_id*, you
can only do that in Web UI. That is, you need to build multiple entries
with different host, but with the same *conn_id*.

The method of setting connection in environment variables can only help set
one connection for each *conn_id*.

(Of course Web UI stores your connections in the metadata database. So if
you really need to avoid manual interventions and need multiple connections
for one single conn_id, hacking into the metadata database may be an
option? even thought not recommended)

Other guys please correct me if I'm wrong in any parts. Thanks.


Regards,
XD

On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br.laird@xxxxxxxxx> wrote:

> Hello -
>
> I'd like to define multiple connections for the `webhdfs_default` conn_id,
> as we have a primary and backup host for the hadoop namenode. This is for
> use with the WebHDFSHook, which queries for connections and seemingly can
> accept a list of them from the database.
>
> ```
>         nn_connections = self.get_connections(self.webhdfs_conn_id)
>         for nn in nn_connections:
> ```
>
> https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L55
>
> I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
> each hostname, however with the CLI and adding the entries to the
> airflow.cfg I get errors that a connection already exists.
>
> It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
> Is this a bug? Or is there a different way to accomplish without manually
> adding via the UI?
>
> Thanks!
> Ben Laird
>