# Lab: OIDC-Protected App on K3s Cluster
> **Date:** 17 January, 2026
> **Series:** Keycloak
> **Focus:** OIDC Authentication — proving identity
> **Environment:** Single-node K3s on a Proxmox Ubuntu Server VM
> Prerequisite: [[K3s Lightweight Kubernetes install]]
---
## What We Are Building
A fully local OIDC authentication stack running inside K3s. When you visit the OIDC-protected app `"whoami.local"`, you get redirected to Keycloak to log in. After authenticating, you land on the `whoami` app which displays your JWT token and all identity claims in plain text.
**The full request flow:**
```
Browser → K3s Traefik → oauth2-proxy → Keycloak (login) → oauth2-proxy → whoami app
```
---
## Why OIDC, Not OAuth2
OIDC is built on top of OAuth2. OAuth2 answers *"is this app allowed?"* — OIDC answers *"who is this user?"*. In this lab we care about identity, so OIDC is the focus. OAuth2 is the underlying plumbing the redirect flow runs on.
| | OAuth2 | OIDC |
| -------------------- | -------------------- | ---------------------- |
| Question answered | Is this app allowed? | Who is this user? |
| Token type | Access Token | ID Token (JWT) |
| Primary use case | API authorization | Login / Authentication |
| Knows user identity? | No | Yes |
---
## Prerequisites
SSH into your VM and confirm:
```bash
# K3s is running
kubectl get nodes
# Expected output:
# NAME STATUS ROLES AGE VERSION
# your-vm Ready control-plane,master 1m v1.x.x+k3s1
```
---
## Step 1 — Deploy Keycloak
> **Important:** Do not use the Bitnami Helm chart. Bitnami moved most images to a legacy repository in August 2025 — images are unsupported and receive no security updates. Use the official Keycloak image from `quay.io` instead.
### 1.1 Create a working directory
```bash
mkdir ~/lab1/keycloak && cd ~/lab1/keycloak
```
### 1.2 keycloak-namespace.yaml
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: keycloak
```
### 1.3 keycloak-deployment.yaml
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
namespace: keycloak
spec:
replicas: 1
selector:
matchLabels:
app: keycloak
template:
metadata:
labels:
app: keycloak
spec:
containers:
- name: keycloak
image: quay.io/keycloak/keycloak:26.3.3
args:
- start-dev
env:
- name: KC_HOSTNAME
value: keycloak.local
- name: KC_HOSTNAME_STRICT
value: "false"
- name: KC_HTTP_ENABLED
value: "true"
- name: KC_PROXY_HEADERS
value: xforwarded
- name: KEYCLOAK_ADMIN
value: admin
- name: KEYCLOAK_ADMIN_PASSWORD
value: admin123
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /realms/master
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
```
> **Data persistence note:** `start-dev` uses an embedded H2 database. All realm config, users, and clients are lost if the pod restarts. For a lab this is acceptable — just be prepared to recreate your realm config if the pod cycles.
**Key decisions explained:**
- `start-dev` — disables the HTTPS requirement and uses an embedded H2 database. Fine for a lab, not for production.
- `KC_HOSTNAME_STRICT: false` — allows Keycloak to respond to requests regardless of the exact hostname match. Required for local dev.
- `KC_PROXY_HEADERS: xforwarded` — tells Keycloak to trust the `X-Forwarded-*` headers Traefik injects. Without this, Keycloak thinks all requests originate from inside the cluster.
- The readiness probe hits `/realms/master` — Kubernetes waits until Keycloak is actually ready before marking the pod healthy.
### 1.4 keycloak-service.yaml
```yaml
apiVersion: v1
kind: Service
metadata:
name: keycloak
namespace: keycloak
spec:
selector:
app: keycloak
ports:
- port: 80
targetPort: 8080
```
Maps port `80` on the service to port `8080` on the container. We do not define the service type here, so this service defaults to "ClusterIP". The ClusterIP service type allows internal communication only (pod to pod).
### 1.5 keycloak-ingress.yaml
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: keycloak-ingress
namespace: keycloak
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
ingressClassName: traefik
rules:
- host: keycloak.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: keycloak
port:
number: 80
```
Tells Traefik to route `keycloak.local` traffic into the Keycloak service. Without this, Keycloak is only reachable from inside the cluster.
### 1.6 Apply and watch
```bash
kubectl apply -f keycloak-namespace.yaml
kubectl apply -f keycloak-deployment.yaml
kubectl apply -f keycloak-service.yaml
kubectl apply -f keycloak-ingress.yaml
# Takes 2-3 minutes — wait for 1/1 Running
kubectl get pods -n keycloak -w
```
---
## Step 2 — Configure Keycloak
Once the pod shows `1/1 Running`, open `http://keycloak.local`.
**Admin credentials:** `admin / admin123`
> The `admin` user only exists in the **master realm**. It is the Keycloak system admin, not a user you log into your apps with. Do not try to use `admin` to log into `whoami.local` — create `testuser` below for that.
### 2.1 Create a Realm
A realm is Keycloak's way of carving out an isolated space for your users, apps, and settings. Think of it as a tenant — everything inside `myrealm` is completely separate from the `master` realm. The master realm is reserved for Keycloak administration only. Any real application you protect should live in its own realm, which is why we create one here before doing anything else.
1. Click the dropdown top-left (shows `master`) → **Create Realm**
2. Name: `myrealm` → **Create**
### 2.2 Create a Client
A client is how Keycloak knows about your application. When oauth2-proxy redirects a user to Keycloak for login, it identifies itself using the client ID. Keycloak checks that this client exists, that the redirect URI matches what was registered, and that the client secret is correct before it will issue any tokens. Without registering a client, Keycloak has no idea who is asking for tokens and will reject the request.
1. Left sidebar → **Clients** → **Create client**
2. Client ID: `whoami-client` | Type: `OpenID Connect` → **Next**
3. Enable **Standard flow** (Authorization Code) → **Next**
4. Valid redirect URIs: `http://whoami.local/oauth2/callback`
5. Web origins: `http://whoami.local` → **Save**
6. **Credentials** tab → copy the **Client secret**
> The client secret is a shared password between Keycloak and oauth2-proxy. It proves to Keycloak that the token request is coming from a legitimate registered client — not an arbitrary application. You will paste this into `oauth2-proxy-deployment.yaml` in Step 3.3.
### 2.3 Create a Test User
This is the actual human user account that will log into `whoami.local`. Keycloak manages user identities — their username, email, password, and any roles or attributes you assign. When the login form appears at `keycloak.local`, Keycloak validates the credentials entered against this user record and, if correct, issues a JWT token containing that user's identity. Without a user in `myrealm`, there is nobody to log in as.
1. Left sidebar → **Users** → **Create new user**
2. Username: `testuser` | Email: `
[email protected]` → **Create**
3. **Credentials** tab → **Set password** → disable **Temporary** toggle
> This is the user you log into `whoami.local` with. Keep this separate in your head from the Keycloak admin account — `admin` exists only in the master realm and cannot log into apps protected by `myrealm`.
---
## Step 3 — Deploy whoami and oauth2-proxy
```bash
mkdir ~/lab1/app && cd ~/lab1/app
```
### 3.1 Generate a cookie secret
```bash
openssl rand -base64 32
# Copy this output — needed in the next file
```
> When a user successfully logs in, oauth2-proxy creates a session cookie and stores it in the user's browser. This cookie is what lets you visit `whoami.local` again without being sent back to the Keycloak login page every single time — oauth2-proxy reads the cookie, confirms the session is valid, and lets you through.
>
> The cookie secret is the key oauth2-proxy uses to encrypt and sign that cookie. Encrypting it means nobody can read its contents if they intercept it. Signing it means if anyone tries to tamper with the cookie — for example, modifying it to impersonate a different user — oauth2-proxy will detect that the signature no longer matches and reject it.
>
> It never leaves the server. Keycloak never sees it. It exists purely between oauth2-proxy and the browser to keep the session secure.
### 3.2 whoami-deployment.yaml
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoami
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: whoami
namespace: default
spec:
selector:
app: whoami
ports:
- port: 80
targetPort: 80
```
`image:traefik/whoami` is an ultra-lightweight container that reflects all incoming HTTP headers back to the browser. It has no authentication logic — it blindly displays whatever headers it receives, which makes JWT claims visible when visiting `whoami.local`, without writing any app code.
### 3.3 oauth2-proxy-deployment.yaml
Replace `YOUR_CLIENT_SECRET` (from Step 2.2) and `YOUR_COOKIE_SECRET` (from Step 3.1) before applying.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: oauth2-proxy
template:
metadata:
labels:
app: oauth2-proxy
spec:
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:latest
args:
- --provider=oidc
- --oidc-issuer-url=http://keycloak.local/realms/myrealm
- --client-id=whoami-client
- --client-secret===YOUR_CLIENT_SECRET==
- --redirect-url=http://whoami.local/oauth2/callback
- --upstream=http://whoami.default.svc.cluster.local
- --http-address=0.0.0.0:4180
- --cookie-secret===YOUR_COOKIE_SECRET==
- --cookie-secure=false
- --email-domain=*
- --pass-authorization-header=true
- --pass-access-token=true
- --set-xauthrequest=true
- --skip-provider-button=true
ports:
- containerPort: 4180
---
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
namespace: default
spec:
selector:
app: oauth2-proxy
ports:
- port: 4180
targetPort: 4180
```
> **Note on oauth2-proxy:** Despite the name, when you pass `--provider=oidc` it runs the full OIDC Authorization Code Flow. The name is a legacy hangover.
**Key args explained:**
| Arg | Purpose |
|---|---|
| `--provider=oidc` | Uses OIDC not plain OAuth2 |
| `--oidc-issuer-url` | Points to Keycloak discovery endpoint — oauth2-proxy fetches all URLs from here automatically |
| `--redirect-url` | Where Keycloak sends the user after login — must match Valid Redirect URIs in Keycloak exactly |
| `--upstream` | Where authenticated requests are forwarded — the whoami service |
| `--cookie-secure=false` | Required since we have no HTTPS in dev |
| `--pass-access-token=true` | Injects the JWT as `X-Auth-Request-Access-Token` header into upstream requests |
| `--set-xauthrequest=true` | Injects `X-Auth-Request-User` and `X-Auth-Request-Email` headers |
| `--skip-provider-button=true` | Skips the intermediate "click to login" page, redirects straight to Keycloak |
> **The redirect-url / Valid Redirect URIs relationship:** `--redirect-url` is what oauth2-proxy tells Keycloak to use when building the authorization request. The Valid Redirect URIs in Keycloak is a security check — Keycloak refuses to redirect anywhere not on that list. Both must match exactly or you get a `redirect_uri_mismatch` error.
### 3.4 ingress.yaml
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: whoami-ingress
namespace: default
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
ingressClassName: traefik
rules:
- host: whoami.local
http:
paths:
- path: /oauth2
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 4180
- path: /
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 4180
```
Both paths point to oauth2-proxy, not directly to whoami. The ingress is evaluated twice across the flow:
- First hit: `GET whoami.local/` matches `/` → oauth2-proxy checks for session cookie, finds none, redirects to Keycloak
- Second hit: `GET whoami.local/oauth2/callback?code=xxxx` (Keycloak's redirect back) matches `/oauth2` → oauth2-proxy exchanges the code for tokens and sets the session cookie
### 3.5 Apply everything
```bash
kubectl apply -f whoami-deployment.yaml
kubectl apply -f oauth2-proxy-deployment.yaml
kubectl apply -f ingress.yaml
kubectl get pods -n default
# Expect: whoami-xxx 1/1 Running, oauth2-proxy-xxx 1/1 Running
```
---
## Step 4 — Networking: /etc/hosts
### How routing outside of the VM works
We will use the local machines `/etc/hosts` file to create a local dns mapping of keycloak.local and whoami.local. You do not need to specify ports in `/etc/hosts`. Port routing is handled entirely by Traefik.
Here is the full chain:
```
Your machine
/etc/hosts: keycloak.local → 192.168.x.x (VM IP)
│
▼
Proxmox bridged networking delivers packet to VM
│
▼
K3s ServiceLB binds Traefik to VM's network interface on port 80
│
▼
Traefik reads Host header → routes to correct service
keycloak.local → Keycloak service
whoami.local → oauth2-proxy service
```
All apps share the same VM IP and port 80 (keycloak and whoami). Traefik distinguishes them by hostname, not port.
### Verify Traefik has an external IP
```bash
kubectl get svc -n kube-system traefik
# Look for EXTERNAL-IP — this is your VM's IP
```
If `EXTERNAL-IP` shows `<pending>`, check:
```bash
# ServiceLB pods should be running
kubectl get pods -n kube-system | grep svclb
```
Also confirm your VM uses **bridged networking** (attached to `vmbr0`), not NAT. With NAT the VM gets an internal IP unreachable from your local machine.
### Add apps to /etc/hosts on your local machine
```bash
# Linux/Mac
sudo nano /etc/hosts
# Add (replace with your actual VM IP from above)
192.168.x.x keycloak.local whoami.local
# Windows: C:\Windows\System32\drivers\etc\hosts
```
All future apps can be added to the same line — they all will share the same VM IP.
### Architecture: How the Files Connect
```
/etc/hosts: whoami.local → VM IP
│
▼
Traefik (ingress.yaml)
Host: whoami.local → oauth2-proxy:4180
│
├── No session cookie
│ │
│ ▼
│ keycloak-ingress.yaml + keycloak-deployment.yaml
│ Browser redirected to keycloak.local/realms/myrealm/...
│ User signs in here — Keycloak issues auth code
│ │
│ ▼
│ Browser GET whoami.local/oauth2/callback?code=xxxx
│ oauth2-proxy exchanges code → tokens → sets session cookie
│ |
└── Valid session cookie
│
▼
oauth2-proxy injects headers:
X-Auth-Request-User
X-Auth-Request-Email
X-Auth-Request-Access-Token
│
▼
whoami-deployment.yaml
Reflects all headers to browser
```
---
## Step 5 — Test the Flow
Open `http://whoami.local` in your browser.
**What should happen:**
1. Immediately redirected to `keycloak.local` login page
2. Log in as `testuser` with the password you set
3. Redirected back to `whoami.local`
4. whoami displays all request headers
**Look for these headers in the whoami output:**
```
X-Auth-Request-User: testuser
X-Auth-Request-Email:
[email protected]
X-Auth-Request-Access-Token: eyJhbGci...
```
### Inspect the JWT
Copy the value from `X-Auth-Request-Access-Token` and paste it into [jwt.io](https://jwt.io). You will see the decoded payload:
```json
{
"sub": "uuid-of-user",
"realm_access": {
"roles": ["default-roles-myrealm"]
},
"email": "
[email protected]",
"preferred_username": "testuser"
}
```
This is the OIDC identity layer in action — the user's identity is cryptographically signed into the token by Keycloak. oauth2-proxy verified the signature using Keycloak's JWKS endpoint before forwarding the request.
---
## Troubleshooting
### user_not_found error in Keycloak logs
```
error="user_not_found", username="admin"
```
You are trying to log into `whoami.local` with the Keycloak admin account. The `admin` user only exists in the master realm — it cannot log into apps protected by `myrealm`. Log in as `testuser` instead.
### Keycloak unreachable from oauth2-proxy
Test internal cluster DNS resolution:
```bash
kubectl run curl --image=curlimages/curl -it --rm -- \
curl http://keycloak.keycloak.svc.cluster.local/realms/myrealm/.well-known/openid-configuration
```
If this fails, the service name or namespace is wrong. Double check `keycloak-service.yaml` is applied and the namespace matches.
### redirect_uri_mismatch error
The `--redirect-url` in `oauth2-proxy-deployment.yaml` does not match the Valid Redirect URIs configured in Keycloak. Both must be exactly `http://whoami.local/oauth2/callback` with no trailing slash differences.
### EXTERNAL-IP pending on Traefik
ServiceLB is not assigning your VM's IP. Check your Proxmox VM is using bridged networking. With NAT networking the VM has no routable IP on your local network and ServiceLB has nothing to bind to.
### Lost realm config after pod restart
The H2 database is ephemeral in `start-dev` mode. Recreate your realm config from Step 2. To prevent this permanently, add a PersistentVolumeClaim mounting `/opt/keycloak/data` — a worthwhile follow-up exercise.
---
## Key Concepts Summary
| Concept | What it means in this lab |
|---|---|
| OIDC Authorization Code Flow | The redirect chain: browser → Keycloak → callback → token |
| JWT | The signed token containing your identity claims |
| Claims | Key-value pairs inside the JWT: email, username, roles |
| JWKS | Keycloak's public keys — used by oauth2-proxy to verify the token signature without calling Keycloak each time |
| Discovery endpoint | `/.well-known/openid-configuration` — Keycloak advertises all its URLs here so oauth2-proxy can configure itself automatically |
| oauth2-proxy | The authentication gate — intercepts all requests, enforces login, injects identity headers |
| Traefik ingress | Routes hostnames to services — all apps share one IP, differentiated by Host header |
---
*Lab environment: K3s single-node on Proxmox Ubuntu VM · Keycloak 26.3.3 · Traefik ingress (built-in K3s)*