[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Multiple hosts for a single connection

Hello -

I'd like to define multiple connections for the `webhdfs_default` conn_id,
as we have a primary and backup host for the hadoop namenode. This is for
use with the WebHDFSHook, which queries for connections and seemingly can
accept a list of them from the database.

        nn_connections = self.get_connections(self.webhdfs_conn_id)
        for nn in nn_connections:

I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
each hostname, however with the CLI and adding the entries to the
airflow.cfg I get errors that a connection already exists.

It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
Is this a bug? Or is there a different way to accomplish without manually
adding via the UI?

Ben Laird