osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Storing a GKE key in our travis build


I got pretty close with docker-in-docker when we were running into
docker-compose issues and wouldn't recommend in it's current form. The
actual changes required wouldn't be hard code-base wise for gke/vanilla
kubernetes. Just need to figure out how to turn it on/off so people can't
run arbitrary code in our GKE (though technically they're already running
arbitrary code in our Travis so 🤷‍♂️).

On Sun, Dec 30, 2018, 2:06 AM Eamon Keane <eamon.keane1@xxxxxxxxx> wrote:

> I think that's a good idea.
>
> There are a few kubernetes-in-docker minikube alternatives although I'm not
> sure if they address the issues of speed and ease of local testing and
> they're still alpha.
>
> https://github.com/kubernetes-sigs/kind (single node for now)
> https://github.com/kubernetes-sigs/kubeadm-dind-cluster (multi-node)
>
> For GKE launching a new cluster takes 150-300 seconds (
> https://kubedex.com/google-gke-vs-azure-aks-automation-and-reliability/).
> Is it the intention to connect to an existing cluster or make one from
> scratch each time?
>
> Using one cluster project-wide for testing should be feasible if it's set
> up correctly (if every commit is namespaced ($branch-$sha) and the first
> step of the script is to delete the namespace if it exists). It should be
> reasonably maintenance-free if set up with node auto-provisioning with a
> cap of say 60 cores depending on how many concurrent builds occur.
>
>
> https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning
>
> On Sun, Dec 30, 2018 at 3:09 AM Daniel Imberman <daniel.imberman@xxxxxxxxx
> >
> wrote:
>
> > Hello all,
> >
> > I wanted to ask about us moving away from minikube for our airflow
> > kubernetes builds.
> >
> > 1. It takes a long time to build
> >
> > Building minikube from scratch every time takes the lionshare of our
> travis
> > testing time. If we can just connect to GKE and launch a cluster in a few
> > seconds, we'll get much faster feedback for whether our changes are
> working
> > or not.
> >
> > 2. It doesn't really mimic real world kubernetes
> >
> > We've run into a few bugs in the wild where our minikube-based testing
> > worked fine but we found issues when performing on a multi-node cluster.
> > Many in the k8s community are even moving away from minikube because of
> > these differences in behavior.
> >
> > 3. It makes local testing a pain
> >
> > Over the next few months I want to start figuring out ways to make
> > iterative local testing of airflow easier. An ideal for k8s-based testing
> > to launch a GKE cluster and just run a script every time we want to test
> > our changes. This would simplify onboarding new devs and reduce the time
> to
> > approve PRs.
> >
> >
> > *Proposal: Create GKE testing that committers can turn on*
> >
> > When users submit PRs to kubernetes, there seems to be a buildbot setting
> > that allows the committers to turn on integration testing on the
> > CloudNative clusters. I'm thinking if all k8s testing is reformatted to
> > assume kubernetes rather than minikube, then users can test as much as
> they
> > want on their own local/GKE clusters and then we can turn on GKE before
> > merging.
> >
> > Google had spoken with us a while back about giving a GKE account for
> > airflow integration, and I wanted to open this up to the community if
> > anyone had strong feelings either way.
> >
>