So that titles a bit of a mouthful - what am I actually going to talk about here? We are currently looking at containerizing a number of our applications, as we have heavily invested into Azure our deployment 'pattern' of choice is using Azure Kubernetes Services (AKS) - there are other ways to host containers in Azure but I'm ignoring those - this seems to be the strategic direction Microsoft are going in and is the most PaaS like of the offerings.
We have taken the decision to expose a little as possible publicly so everything we deploy is generally only accessible via the private express route connection. This means the surface to be attacked is much less than the normal public facing deployments that are the general default for a lot of the PaaS services. This means we are deploying the app into a VNET attached service - this is a relatively new offering which only went GA a few months ago.
In this post I'll take you through the steps we followed to set up the AKS and then protect that simple app (one just using http and no authentication) to one using https and azure AD based authentication.
The first step of doing this is to deploy an AKS - this is a relatively straightforward operation so I'll just skim over the details without a lot of explanation
The main screen of interest are:
The basic of what you are provisioning here:
And the networking screen here - the other tabs don't really matter for the example case I'm running here.
So once you are happy go ahead and create that - this can take a while ~45 mins in recent attempts so be aware of that. In our case we are always deploying to pre-defined networks as we have a complex setup with user defined routes and firewall appliances - this may or may not be true for you.
Once you have that up and running you'll have a number of azure resources created - split between the resource group you specified when you added the objects and a 'magically' created one prefixed with MC_ that actually contains the azure resources underlying the service - i.e. the VM's, load balancers and the like.
OK now we have this infrastructure in place the basic app can be deployed to it, i won't include the details here of our app as its in house developed - but it could be any simple application that is presented from a http:// endpoint - so for example it could just be a simple apache server with the default homepage. If you want to just test this out then the aks-helloworld app can be used (helm can install this).
After that stage you should have a site you can visit that is accessible from you client machine - it has no certificate though and no authentication in front of it - it's accessible to anyone that knows the url.
So now lets move ahead and protect this - we'll start out with just hiding it behind a https endpoint
I'm going to make the assumption now that you have copied the kube connection config files down from the portal to a client machine where you have the kubectl command line installed and you have also downloaded the helm executable (helm is a helper kind of tool that lets you easiest instantiate images without having to write loads of horrible yaml)
So with that said this is the first step - install an nginx reverse proxy against an azure internal load balancer. This step actually took me ages to get right as some of the docs seem to be wrong and many blogs i found explain how to do this in a way that didn't work for me - this is the command i ran
helm install stable/nginx-ingress --namespace kube-system --name general-ingress --set rbac.create=false --set rbac.createRole=false --set rbac.createClusterRole=false -f rich.yaml
The contents of rich.yaml being
controller:
service:
loadBalancerIP: 10.x.x.x
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
The 10.x.x.x address should be one that is free in Azure, if you try and reserve it in advance as some notes suggest it doesn't work properly. This address must be in the same subnet as the host machines but outside of the range of (default 30 per machine) that are reserved. You should be able to browse the subnet to find the next available ip to be used.
Once this is provisioned you should assign a dns label against this ip address in wherever your dns is managed as you'll need to reference to this name later on in further config.
After that is created we need to add a couple of components to enable the https ingress.
First up a certifcate manager
helm install stable/cert-manager --set ingressShim.defaultIssuerName=letsencrypt-staging --set ingressShim.defaultIssuerKind=ClusterIssuer --set rbac.create=false --set serviceAccount.create=false
Then we need to define a clusterissuer
kubectl apply -f clusterissuer.yaml
where clusterissuer.yaml looks like this
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: youremail@domain
privateKeySecretRef:
name: letsencrypt-staging
http01: {}
Then we need to create the certificate part
kubectl apply -f certificates.yaml
where certificates.yaml contains the following
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: tls-secret
spec:
secretName: tls-secret
dnsNames:
- dns-domain-you-defined-earlier
acme:
config:
- http01:
ingressClass: nginx
domains:
- dns-domain-you-defined-earlier
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
The final step is then to create an ingress that maps to the simple app to the load balancer endpoint we created earlier
kubectl apply -f test.yaml
where test.yaml contains this
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- dns-domain-name-we-created-earlier
rules:
- http:
paths:
- path: /
backend:
serviceName: aks-helloworld
servicePort: 80
So what happens is when we access the site https://dns-domain-name-we-created-earlier (which has the dummy cert issued by letsencrypt) the path directive near the bottom of that last yaml file maps / (i.e. no path at the end of the url) to a service called aks-helloworld running on a kubernetes internal port of 80 - this content is then served up as if it came from the https version of our site.
So we now have a https version of our site (albeit with a dummy cert which causes warnings)
So with no changes to our actual app we now support https access - in fact this model could mean we have multiple sites accessed via different paths/urls all protected by the single nginx reverse proxy - this is quite neat (well it is when you get your head around all the new concepts and what it gives you).
For reference a lot of the above steps are taken directly from here - which explains it better than i do and the site also looks nicer than mine - but they do have a bigger budget :-)
So one requirement fulfilled now we have the second one - how to add Azure AD authentication and protect the site?
Well again this is reasonable simple - first i need to register an application in Azure AD - simple demo one shown below
First browse to app registrations
give it a name and a sign on url that matches the dns name we created earlier
Then create a 'password' as shown below
This only appears once so be sure to grab it
Along with that password you'll need the new application id that has been created - see screenshot below for reference to that
Be sure to set the reply url is set to https://yourdnsnamefromearlier/oauth2/callback
Thats the Azure AD part covered now we need to integrate to this from kubernetes
The example i followed to help me set this up is here https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/auth/oauth-external-auth/oauth2-proxy.yaml
It uses a container image from colemickens (thanks to cole for that)
I deploy this via
kubectl apply -f oauth2-deployment.yaml
My deployment file ended up looking like this
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: oauth2-proxy
spec:
replicas: 3
selector:
matchLabels:
app: oauth2-proxy
template:
metadata:
labels:
app: oauth2-proxy
spec:
containers:
- args:
- --provider=azure
- --email-domain=your email domain here (contoso.com for example)
- --upstream=file:///dev/null
- --http-address=0.0.0.0:4180
- --azure-tenant=tenant id you can find from the azure portal
env:
- name: OAUTH2_PROXY_CLIENT_ID
value: application id we got from portal earlier
- name: OAUTH2_PROXY_CLIENT_SECRET
value: value-we-captured from azure ad portal earlier for 'password'
- name: OAUTH2_PROXY_COOKIE_SECRET
value: O0flmLmQWStGoSHyuLjbMw== # value of: python -c 'import os,base64; print base64.b64encode(os.urandom(16))'
image: docker.io/colemickens/oauth2_proxy:latest
imagePullPolicy: Always
name: oauth2-proxy
ports:
- containerPort: 4180
protocol: TCP
kubectl apply -f proxyservice.yaml
the yaml file containing
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
spec:
ports:
- name: http
port: 4180
protocol: TCP
targetPort: 4180
selector:
app: oauth2-proxy
Finally i create an ingress point into this as its needs to be reachable for my clients to authenticate against - code for that is
kubectl apply -f oauthingress.yaml
with the yaml file containing
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: oauth2-proxy-ingress
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
spec:
tls:
- hosts:
- dnsdomainnameicreatedearlier
rules:
- host: dnsdomainnameicreatedearlier
http:
paths:
- path: /oauth2
backend:
serviceName: oauth2-proxy
servicePort: 4180
At this point you should have a site you can visit at https://yoursite/oauth2/start - ths should take you through the azure AD authentication process you have defined for you site (including MFA or whatever you defined)
Whats missing now is the glue to make the normal application site be protected by oauth2 - so we just need to make those changes
To do that all we need to do is add the following 2 lines (shown in orange)to the test.yaml file from earlier
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-url: "http://oauth2-proxy.default.svc.cluster.local:4180/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "http://yourdnsnamehere/oauth2/start"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- dns-domain-name-we-created-earlier
rules:
- http:
paths:
- path: /
backend:
serviceName: aks-helloworld
servicePort: 80
This is applied via
kubectl apply -f test.yaml
Now you'll notice that first line looks a little odd - it's referring to an internal kubernetes name - it should (in my view) work wth the normal dns name but for whatever reason it doesn't. The second line has the normal exposed dns name - in fact it has to have this as its what the client access will get redirected to if they dont already have a token.
Also note that prior to version 1.11.2 the internal kubernetes name above didn't 100% work - it seemed to fail maybe 25% of the time with random could not connect errors - since 1.11.2 it seems to be 100% fine.
So after all that if i browse to https://mydnsname i'll get prompted to authenticate against Azure AD - if i already have a token i go straight in.
So my very basic http://site is now encrypted on the wire with https (with dummy cert) but is also protected by me having to have passed Azure AD authentication.
The example above was for an internally deployed AKS but there is no reason the same concept can't be used to protect a public endpoint to with a 'standard' AKS deployment.
There were a whole load of new concepts i picked up here setting this up (2 weeks ago i didn't even know how to pronounce kubernetes) - hopefully this post can also help people out who are just starting with some of the basics of how a site can be protected.
Getting the above actually working was actually very difficult - the steps in the end were relatively easy but working out the load balancer issue at the start and the bug with older kubernetes versions and networking actually made this really hard.
If you are trying to do something similiar to the above make sure you are on the latest kubernetes version to avoid tearing your hair out in frustration.....
The picture at the start kind of reflects my progress in getting this work done......
This is very nice article which shows the complete end to end setup. Thankyou
ReplyDelete