osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Multiple hosts for a single connection


Hello -

I'd like to define multiple connections for the `webhdfs_default` conn_id,
as we have a primary and backup host for the hadoop namenode. This is for
use with the WebHDFSHook, which queries for connections and seemingly can
accept a list of them from the database.

```
        nn_connections = self.get_connections(self.webhdfs_conn_id)
        for nn in nn_connections:
```
https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L55

I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
each hostname, however with the CLI and adding the entries to the
airflow.cfg I get errors that a connection already exists.

It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
Is this a bug? Or is there a different way to accomplish without manually
adding via the UI?

Thanks!
Ben Laird