Optimizing Oauth2-Proxy Authentication in K8s
Rodrigo Zuolo
September 14, 2024
September 14, 2024
At SafeBoda, we are always concerned about the continuity and security of our services. One crucial component that ensures the protection of our ingresses is Google OAuth authorization, which we leverage as much as possible. Extensive usage of these access control interfaces can result in a burden of extra hours of administration and redundant systems that could be implemented in a leaner architecture.
This article will present the steps we used in our staging K8s cluster to optimize the ingress authentication infrastructure that relies on Google OAuth. The overall result is an ecosystem that is easier to maintain, demands fewer computing resources, and delivers the desired level of access controls.
This article assumes that you already understand the meaning and purpose of the oauth2-proxy. If that is not the case, in a few words, the so-referred proxy is an nginx deployment that articulates users’ authorization via Open Authorization Protocol, and it is compatible with the most popular providers (Google, Github, Keycloak, etc.). In our use case, the Google API authorization is used with an Ingress-Nginx Controller and OAuth2-Proxy.
In a nutshell, we use the OAuth2-Proxy to interface with many WEB-UI-oriented services in the staging environment. However, some of these services are meant to be accessible by the same set of users. We take, for example, the scenario where a service A and a service B are running on the namespace testing, and they are both accessed by users torricelli@safeboda.com, volta@safeboda.com, and pasteur@safeboda.com. In that case, we could deploy an OAuth2-Proxy instance for the ingress of service A and another OAuth2-Proxy instance for the ingress of service B. Each proxy instance would allow those same users to access the backend web application after being processed via the OAuth authorization process.
In addition to the situation where services are in the same namespace, we could have a third application - service C - running in the namespace playground. This service C is also accessed by the same group of users as service A and B. Following a typical deployment of the OAuth2-Proxy, we would have something similar to the illustration in Figure 1.
Figure 1: Ingresses services sharing the same users’ access lists.
Given that ingresses for services A, B, and C authorize a shared list of users, deploying a single instance of OAuth2-Proxy is a smarter and more efficient setup. This single instance can be running in an isolated namespace. In this use case, the namespace will be referred to as oauth. The deployment of this OAuth2-Proxy must contain the list of the users that are going to be allowed, and its manifest would look like the following:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: sb-oauth2-merged-proxy
name: sb-oauth2-merged-proxy
namespace: oauth
spec:
replicas: 1
selector:
matchLabels:
k8s-app: sb-oauth2-merged-proxy
template:
metadata:
labels:
k8s-app: sb-oauth2-merged-proxy
spec:
containers:
- args:
- --provider=google
- --upstream=file:///dev/null
- --http-address=0.0.0.0:8843
- --authenticated-emails-file=/etc/config/users.dat
env:
- name: OAUTH2_PROXY_CLIENT_ID
value: <SOME_ID_HERE>
- name: OAUTH2_PROXY_CLIENT_SECRET
value: <SOME_SECRET_GOES_HERE>
- name: OAUTH2_PROXY_COOKIE_SECRET
value: <A_COOKIE_IS_HERE>
image: quay.io/oauth2-proxy/oauth2-proxy:latest
imagePullPolicy: Always
name: oauth2-proxy
ports:
- containerPort: 8843
protocol: TCP
volumeMounts:
- name: sb-users
mountPath: /etc/config/users.dat
subPath: users.dat
volumes:
- name: sb-users
configMap:
name: sb-users
---
apiVersion: v1
data:
users.dat: |
torricelli@safeboda.com
pasteur@safeboda.com
volta@safeboda.com
kind: ConfigMap
metadata:
name: sb-users
namespace: oauth
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: sb-oauth2-merged-proxy
name: sb-oauth2-merged-proxy
namespace: oauth
spec:
ports:
- name: http
port: 8843
protocol: TCP
targetPort: 8843
selector:
k8s-app: sb-oauth2-merged-proxy
Next comes the ace in the hole: The Kubernetes external service. In K8s, we can define a service that runs on a specific namespace, but in reality, it redirects to any other service (in an external namespace) as long as the Fully Qualified Domain Name (FQDN) is used correctly. This holds the same concepts as name aliases on a name translation system. So, if we wanted to create a service, listening on the testing namespace that redirects to a service running in the namespace oauth, we deploy a manifest like the following.
kind: Service
apiVersion: v1
metadata:
namespace: testing
name: sb-oauth2-merged-proxy
spec:
type: ExternalName
externalName: sb-oauth2-merged-proxy.oauth.svc.cluster.local
ports:
- protocol: TCP
port: 8843
We follow suit for the playground namespace, and then both services point to the oauth namespace, where our single OAuth2-Proxy instance is running.
kind: Service
apiVersion: v1
metadata:
namespace: playground
name: sb-oauth2-merged-proxy
spec:
type: ExternalName
externalName: sb-oauth2-merged-proxy.oauth.svc.cluster.local
ports:
- protocol: TCP
port: 8843
After that, we can create individual ingresses for each service in their respective namespaces, but we redirect the authorization to the external service's FQDN. That, in turn, points to the OAuth2-Proxy in the oauth namespace. In our particular case, we are using ingresses based upon the Ingress-Nginx Controller; thus, the ingress used for interfacing service A, in the testing namespace would have a manifest as follows:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
name: service-a-backend
namespace: testing
spec:
ingressClassName: nginx
rules:
- host: service-a-testing.safeboda.com
http:
paths:
- backend:
serviceName: service-a
servicePort: 80
path: /
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: service-a-oauth
namespace: testing
spec:
ingressClassName: nginx
rules:
- host: service-a-testing.safeboda.com
http:
paths:
- backend:
serviceName: sb-oauth2-merged-proxy
servicePort: 8843
path: /oauth2
tls:
- hosts:
- service-a-testing.safeboda.com
secretName: our-testing-tls
Note that in the snippet above, we split the ingress into two parts. The first ingress captures the external requests and redirects them to the authentication phase. The second ingress receives the OAuth and forwards it to the service sb-oauth2-merged-proxy, which in turn is interfaced by an external name: sb-oauth2-merged-proxy.oauth.svc.cluster.local. Once redirected, the authorization is duly processed by the OAuth2-Proxy, which returns to the original caller to resume the flow in the event of success. At this point, the backend of service A (service-a port 80) is the final traffic destination.
Suppose this ingress is repeated to service B and service C, or any other service in any namespace. In that case, it is possible to provide OAuth2 controlled access to the list of the users configured in our users.dat configmap. This leaner architecture is depicted in Figure 2.
Figure 2: Ingresses services redirecting to a single OAuth2 proxy.
With this arrangement, the ingresses for services A, B, and C will have the users torricelli@safeboda.com, volta@safeboda.com, and pasteur@safeboda.com ready to be authorized using a single instance of OAuth2-Proxy, and… voilà!
All in all, maintaining this authorization layer becomes more accessible with fewer resource demand. So, if a shared group of users access multiple services via ingress, we can leverage this setup to make our cloud engineers' routine easier.
I hope this has been useful.
Thanks for your reading, and enjoy k8s external names.
September 14, 2024