osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

WebHdfsSensor doesn't support HDFS HA


Hi all,

We've been using WebHdfsSensor happily to sensor the state of upstream
tasks outputting to HDFS except when there is a namenode switch. I've
opened https://issues.apache.org/jira/browse/AIRFLOW-2901 to discuss the
HDFS HA support.

There are two solutions that I can see,

1. use pyarrow.hdfs which has HA support
2. allow user to configure a list of namenodes

WDYT ?

Thanks,
Manu Zhang