osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Discuss] FLIP-26 - SSL Mutual Authentication


FYI: The 1.6 docs reflect the setup where internal and external SSL are
separately configured, and where internal SSL uses client authentication.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-ssl.html

On Mon, Aug 13, 2018 at 8:54 AM, Stephan Ewen <sewen@xxxxxxxxxx> wrote:

> Sounds good, Eron!
>
> Please go ahead...
>
> On Sat, Jul 28, 2018 at 1:33 AM, Eron Wright <eronwright@xxxxxxxxx> wrote:
>
>>  As an update to this thread, Stephan opted to split the internal/external
>> configuration (by providing overrides for a common SSL configuration):
>> https://github.com/apache/flink/pull/6326
>>
>> Note that Akka doesn't support hostname verification in its 'classic'
>> remoting implementation (though the new Artery implementation apparently
>> does), and such verification wouldn't apply to the client certificate
>> anyway.   So the reality is that one should use a limited truststore
>> (never
>> the system truststore) for Akka communication.
>>
>> On the question of routing external communication thru the YARN resource
>> proxy or Mesos/DCOS admin router, the value proposition is:
>> a) simplifies service discovery on the part of external clients,
>> b) permits single sign-on (SSO) be delegating authentication to a central
>> authority,
>> c) facilitates access from outside the cluster, via a public address.
>> The main challenge is that the Flink client code must support a more
>> diverse array of authentication methods, e.g. Kerberos when communicating
>> with the YARN proxy.
>>
>> Given #6326, the next steps would be (unordered):
>> a) create an umbrella issue for the overall effort
>> b) dive into the authorization work for external communication
>> c) implement auto-generation of a certificate for internal communication
>> d) implement TLS on queryable state interface (FLINK-5029)
>>
>> I'll take care of (a) unless there is any objection.
>> -Eron
>>
>>
>> On Sun, May 13, 2018 at 5:45 AM Stephan Ewen <ewenstephan@xxxxxxxxx>
>> wrote:
>>
>> > Throwing in some more food for thought:
>> >
>> > An alternative to the above proposed separation of internal and external
>> > SSL would be the following:
>> >
>> >   - We separate channel encryption and authentication
>> >   - We use one common SSL layer (internal and external) that is in both
>> > cases only responsible for establishing an encrypted connection
>> >   - Authentication / authorization internally is done by SASL with
>> > username/password or shared secret.
>> >   - Authentication externally must be through a proxy and authorization
>> > based on a validating HTTP headers set by the proxy, as discussed
>> above..
>> >
>> > Advantages:
>> >   - There is only one certificate needed, which could also be shared
>> across
>> > applications
>> >   - One or two lines in the config authenticate and authorize internal
>> > communication
>> >   - One could possibly still fall back to the other mode by skipping
>> >
>> > Open Questions / Disadvantages
>> >   - Given that hostname verification during SSL handshake is not
>> possible
>> > in many setups, the encrypted channel is vulnerable to man-in-the-middle
>> > attacks without mutual authentication. Not sure how serious that is,
>> > because it would need an attacker to have compromise network nodes of
>> the
>> > cluster already. is that not a universal issue in the K8s world?
>> >
>> > This is anyways a bit hypothetical, because as long as we have akka
>> beneath
>> > the RPC layer, we cannot go with that approach.
>> >
>> > However, if we want to at least keep the door open towards something
>> like
>> > that in the future, we would need to set up configuration in such a way
>> > that we have a "common SSL" configuration (keystore, truststore, etc.)
>> and
>> > internal/external options that override those. That would anyways be
>> > helpful for backwards compatibility.
>> >
>> > @Eron - what are your thoughts on that?
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Sun, May 13, 2018 at 1:40 AM, Stephan Ewen <ewenstephan@xxxxxxxxx>
>> > wrote:
>> >
>> > > Thank you for bringing this proposal up. It looks very good and we
>> seem
>> > to
>> > > be thinking along very similar lines.
>> > >
>> > > Below are some comments and thoughts on the FLIP.
>> > >
>> > > *Internal vs. External Connectivity*
>> > >
>> > > That is a very helpful distinction, let's build on that.
>> > >
>> > >   - I would suggest to treat eventually all communication coming
>> > > potentially from users as external, meaning Client-to-Dispatcher,
>> > > Client-to-JobManager (trigger savepoint, change parallelism, ...), Web
>> > UI,
>> > > Queryable State.
>> > >
>> > >   - That leaves communication that is only between
>> > JobManager/TaskManager/
>> > > ResourceManager/Dispatcher/HistoryServer as internal.
>> > >
>> > >   - I am somewhat operating under the assumption that all external
>> > > communication will eventually be HTTP/REST. That works best with many
>> > > setups and is the basis for using service proxies that
>> > > handle  authentication/authorization.
>> > >
>> > >
>> > > In Flink 1.5 and future versions, we have the following update there:
>> > >
>> > >   - Akka is now strictly internal connectivity, the client (except
>> legacy
>> > > client) do not use it any more.
>> > >
>> > >   - The Blob Server will move to purely internal connectivity in Flink
>> > > 1.6, where a POST of a job to the Dispatcher has the jars and the
>> > JobGraph.
>> > > That is important for Kubernetes setups, where exposing the BlobServer
>> > and
>> > > querying the blob port causes quite some friction.
>> > >
>> > >   - Treating queryable state as "internal connectivity" is fine for
>> now.
>> > > We should treat it as "external" connectivity in the future if we
>> move it
>> > > to HTTP/REST.
>> > >
>> > >
>> > > *Internal Connectivity and SSL Mutual Authentication*
>> > >
>> > > Simply activating SSL mutual authentication for the internal
>> > communication
>> > > is a really low hanging fruit.
>> > >
>> > > Activating client authentication for Akka, network stack Netty (and
>> Blob
>> > > Server/Client in Flink 1.6) should require no change in the
>> > configurations
>> > > with respect to Flink 1.4. All processes are, with respect to internal
>> > > communication, simultaneously server and client endpoints. Because of
>> > that,
>> > > they already need KeyStore and TrustStore files for SSL handshakes,
>> where
>> > > the TrustStore needs to trust the KeyStore Certificate.
>> > >
>> > > I personally favor the suggestion made to have a script that
>> generates a
>> > > self-signed certificate and adds it to "conf" and updates the
>> > > configuration. That should be picked up by the Yarn and Mesos clients
>> > > anyways.
>> > >
>> > >
>> > > *External Connectivity*
>> > >
>> > > There is a huge surface area and I think we need to give users a way
>> to
>> > > plug in their own tools.
>> > > From what I see (and after some discussions with Patrick and Gary) I
>> > think
>> > > it makes sense to look at proxies in a broad way, similar to the
>> approach
>> > > Eron outlined.
>> > >
>> > > The basic approach could be like that:
>> > >
>> > >   - Everything goes through HTTPS, so the proxy can work with HTTP
>> > headers.
>> > >   - The proxy handles authentication and possibly authorization. The
>> > proxy
>> > > adds some header, for example a user name, a group id, an
>> authorization
>> > > token.
>> > >   - Flink can configure an implementation of an 'authorizer' or
>> validator
>> > > on the headers to decide whether the request is valid.
>> > >
>> > >   - Example 1: The proxy does authentication and adds the user name /
>> > > group as a header. The the Flink-side authorizer simply checks whether
>> > the
>> > > name is in the config (simple ACL-style) scheme.
>> > >   - Example 2: The proxy adds an JSON Web Token and the authorizer
>> > > validates that token.
>> > >
>> > > For secure connections between the Proxy and the Flink Endpoint I
>> would
>> > > follow Eron's suggestion, to use separate KeyStores and TrustStores
>> than
>> > > for internal communication.
>> > >
>> > > For Yarn and Mesos, I would like to see if we could handle those
>> again as
>> > > a special case of the proxies above:
>> > >   - DCOS Admin Router forwards the user authentication token, so that
>> > > could be another authorizer implementation.
>> > >   - In YARN we could see if can implement the IP filter via such an
>> > > authorizer.
>> > >
>> > >
>> > > *Hostname Verification*
>> > >
>> > > For internal communication, and especially on dynamic environments
>> like
>> > > Kubernetes, it is very hard to work with certificates and have
>> hostname
>> > > verification on.
>> > >
>> > > If we assume internal communication works strictly with a shared
>> secret
>> > > certificate and with client authentication, does hostname verification
>> > > actually still add security in that particular setup? My understanding
>> > was
>> > > that hostname verification is important to not have some valid
>> > certificate
>> > > presented, but the one bound to the server you want to talk to. If we
>> > have
>> > > anyways one trusted certificate only, isn't that already implied?
>> > >
>> > > On the other hand, it is still possible (and potentially valuable) for
>> > > users in standalone mode to use keystores and truststores from a PKI,
>> in
>> > > which case there may still be an argument in favor of hostname
>> > verification.
>> > >
>> > > On Thu, May 10, 2018, 02:30 Eron Wright <eronwright@xxxxxxxxx> wrote:
>> > >
>> > >> Hello,
>> > >>
>> > >> Given that some SSL enhancement bugs have been posted lately, I took
>> > some
>> > >> time to revise FLIP-26 which explores how to harden both external and
>> > >> internal communication.
>> > >>
>> > >>
>> > https://cwiki.apache.org/confluence/pages/viewpage.action?
>> pageId=80453255
>> > >>
>> > >> Some recent related issues:
>> > >> - FLINK-9312 - mutual auth for intra-cluster communication
>> > >> - FLINK-5030 - original SSL feature work
>> > >>
>> > >> There's also some recent discussion of how to use Flink SSL
>> effectively
>> > in
>> > >> a Kubernetes environment.   The issue is about hostname verification.
>> > The
>> > >> proposal that I've put forward in FLIP-26 is to not use hostname
>> > >> verification for intra-cluster communication, but rather to rely in a
>> > >> cluster-internal certificate and a truststore consisting only of that
>> > >> certificate.   Meanwhile, a new "external" certificate would be
>> > >> configurable for the web/api endpoint and associated with a
>> well-known
>> > DNS
>> > >> name as provided by a K8s Service resource.
>> > >>
>> > >> Stephan is this in-line with your thinking re FLINK-9312?
>> > >>
>> > >> Thanks
>> > >> Eron
>> > >>
>> > >
>> >
>>
>
>