osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Multiple hosts for a single connection


Hi Ash -

That'd be perfect. Since conn_id isn't a primary key (since we can add via
the UI), I agree that it'd be great to have the same functionality from the
CLI with a flag. I can hopefully do this and submit a PR if folks would
find it useful!

Ben

On Tue, Aug 7, 2018 at 10:28 AM, Ash Berlin-Taylor <ash@xxxxxxxxxx> wrote:

> Hmm yes, it appears that the `airflow connections` CLI doesn't let you
> create multiple connections of the same conn_id. What the WebUI can do the
> CLI should be able to do also! It should allow that in some way (behind a
> `--allow-multiple` flag perhaps? I can see an argument for not allowing set
> multiple by default as it is often not what people want.)
>
> -ash
>
> > On 7 Aug 2018, at 15:17, Deng Xiaodong <xd.deng.r@xxxxxxxxx> wrote:
> >
> > Hi Ben,
> >
> > If you would like to set multiple connections with the same *conn_id*,
> you
> > can only do that in Web UI. That is, you need to build multiple entries
> > with different host, but with the same *conn_id*.
> >
> > The method of setting connection in environment variables can only help
> set
> > one connection for each *conn_id*.
> >
> > (Of course Web UI stores your connections in the metadata database. So if
> > you really need to avoid manual interventions and need multiple
> connections
> > for one single conn_id, hacking into the metadata database may be an
> > option? even thought not recommended)
> >
> > Other guys please correct me if I'm wrong in any parts. Thanks.
> >
> >
> > Regards,
> > XD
> >
> > On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br.laird@xxxxxxxxx> wrote:
> >
> >> Hello -
> >>
> >> I'd like to define multiple connections for the `webhdfs_default`
> conn_id,
> >> as we have a primary and backup host for the hadoop namenode. This is
> for
> >> use with the WebHDFSHook, which queries for connections and seemingly
> can
> >> accept a list of them from the database.
> >>
> >> ```
> >>        nn_connections = self.get_connections(self.webhdfs_conn_id)
> >>        for nn in nn_connections:
> >> ```
> >>
> >> https://github.com/apache/incubator-airflow/blob/master/
> airflow/hooks/webhdfs_hook.py#L54-L55
> >>
> >> I can manually add 2 entries for `webhdfs_default` in the Web UI, one
> for
> >> each hostname, however with the CLI and adding the entries to the
> >> airflow.cfg I get errors that a connection already exists.
> >>
> >> It seems WebHDFSHook doesn't support reading multiple hosts from one
> entry.
> >> Is this a bug? Or is there a different way to accomplish without
> manually
> >> adding via the UI?
> >>
> >> Thanks!
> >> Ben Laird
> >>
>
>