# Troubleshooting: oauth2-proxy CrashLoopBackOff After Server Reboot
> **Date:** 03 July, 2026
> **Series:** Keycloak + OIDC on K3s
> **Type:** Troubleshooting
> **Related:** [[Keycloak (OIDC) on K3s]]
---
## What Happened
After a server reboot, the `oauth2-proxy` pod entered a `CrashLoopBackOff` state. The `whoami` pod was still running, but the app was inaccessible because oauth2-proxy — the authentication gate — could not start. The root cause was Keycloak's embedded H2 database being wiped on pod restart, taking the `myrealm` realm configuration with it. Without the realm, oauth2-proxy could not complete OIDC discovery on startup and failed immediately.
---
## Why This Happens
Keycloak in `start-dev` mode uses an embedded H2 database stored inside the container's filesystem. When a pod restarts, that filesystem is destroyed and recreated fresh from the image. Any configuration created at runtime — realms, clients, users — lives only in that temporary filesystem and does not survive a restart.
This is expected behaviour in `start-dev` mode and is fine for a lab environment, but it does mean that any pod restart requires Keycloak configuration to be recreated manually.
oauth2-proxy is directly affected because it performs OIDC discovery at startup — it calls Keycloak's discovery endpoint to find all the URLs it needs. If the realm does not exist, Keycloak returns a `404` and oauth2-proxy fails to initialise, entering `CrashLoopBackOff` as Kubernetes repeatedly tries to restart it.
---
## Reconnaissance Steps
This is the exact sequence used to diagnose the issue, and the recommended approach any time a pod is behaving unexpectedly.
### Step 1 — Check pod status
```bash
kubectl get pods -n default
```
Output showed oauth2-proxy in `CrashLoopBackOff`:
```
NAME READY STATUS RESTARTS AGE
oauth2-proxy-7c9874867b-7z249 0/1 CrashLoopBackOff 4730 (3m17s ago) 29d
whoami-866c4f9b89-ffn4z 1/1 Running 1 (16d ago) 30d
```
`CrashLoopBackOff` means the container is starting, crashing immediately, and Kubernetes is backing off before trying again. The high restart count (4730) indicates this has been happening for an extended period — consistent with a reboot that happened days or weeks ago.
### Step 2 — Describe the failing pod
```bash
kubectl describe pod oauth2-proxy-7c9874867b-7z249
```
The events section at the bottom of the output confirmed the container was repeatedly being pulled and restarted:
```
Normal Pulling 3m39s kubelet Pulling image "quay.io/oauth2-proxy/oauth2-proxy:latest"
Warning BackOff 2m27s kubelet Back-off restarting failed container oauth2-proxy
```
`describe` is useful for identifying resource issues, image pull failures, and scheduling problems. In this case it confirmed the container itself was the problem, not the node or scheduling — pointing toward a runtime error inside the container.
### Step 3 — Read the logs from the previous crash
Because the container was crashing before it could stay running, `kubectl logs` on its own may return nothing or very little. The `--previous` flag reads logs from the last crashed instance:
```bash
kubectl logs oauth2-proxy-7c9874867b-7z249 --previous
```
Output:
```
[2026/07/03 20:27:22] [provider.go:55] Performing OIDC Discovery...
[2026/07/03 20:27:22] [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: could not create provider data: error building OIDC ProviderVerifier: could not get verifier builder: error while discovery OIDC configuration: failed to discover OIDC configuration: unexpected status "404": {"error":"Realm does not exist"}
```
This is the key line:
```
unexpected status "404": {"error":"Realm does not exist"}
```
oauth2-proxy attempted to contact Keycloak's OIDC discovery endpoint for `myrealm` and received a 404 — the realm did not exist. Keycloak was running, but its configuration had been wiped. Root cause confirmed.
---
## The Fix
### 1. Log into Keycloak admin
Open `http://keycloak.local` and sign in with `admin / admin123`.
### 2. Recreate the realm
- Click the dropdown top-left (shows `master`) → **Create Realm**
- Name: `myrealm` → **Create**
### 3. Recreate the client
- Left sidebar → **Clients** → **Create client**
- Client ID: `whoami-client` | Type: `OpenID Connect` → **Next**
- Enable **Standard flow** → **Next**
- Valid redirect URIs: `http://whoami.local/oauth2/callback`
- Web origins: `http://whoami.local` → **Save**
- **Credentials** tab → copy the new **Client secret**
### 4. Update oauth2-proxy with the new client secret
The client secret generated for `whoami-client` will be different from the previous one. Update `oauth2-proxy-deployment.yaml` with the new value:
```yaml
- --client-secret=YOUR_NEW_CLIENT_SECRET
```
Apply the updated manifest:
```bash
kubectl apply -f ~/lab1/app/oauth2-proxy-deployment.yaml
```
### 5. Recreate the test user
- Left sidebar → **Users** → **Create new user**
- Username: `testuser` | Email: `
[email protected]` → **Create**
- **Credentials** tab → **Set password** → disable **Temporary** toggle
### 6. Confirm oauth2-proxy recovers
```bash
kubectl get pods -n default -w
```
The pod should move from `CrashLoopBackOff` to `Running` within a minute or two once Keycloak's discovery endpoint is responding correctly for `myrealm`.
---
## Confirming the Fix
Once oauth2-proxy shows `1/1 Running`, verify the full flow end-to-end:
```bash
# Confirm all pods are healthy
kubectl get pods -n default
```
Then open `http://whoami.local` in your browser and confirm the Keycloak login page appears.
---
## Confirming Keycloak OIDC Discovery
Run this from your **local machine** to confirm Keycloak is serving the realm correctly. This works because `keycloak.local` is in your local machine's `/etc/hosts`:
```bash
curl -s http://keycloak.local/realms/myrealm/.well-known/openid-configuration | jq .
```
A healthy response will return a large JSON document containing all of Keycloak's OIDC endpoints. If you see `{"error":"Realm does not exist"}` the realm was not recreated correctly — go back to Step 2 of the fix.
> Run this from your local machine, not the VM. The `/etc/hosts` entry for `keycloak.local` lives on your local machine since that is where the browser is. The VM has no hostname entry for `keycloak.local` and the command will fail from there. If you need to run the check from the VM itself, use the IP address directly instead: `curl -s http://192.168.x.x/realms/myrealm/.well-known/openid-configuration | jq .`
---
## Preventing This in the Future
The underlying cause is that Keycloak's H2 database does not persist across pod restarts. There are two ways to address this depending on how far you want to go.
**Option 1 — Add a PersistentVolumeClaim (recommended next step)**
Mounting a persistent volume to Keycloak's data directory means the H2 database survives pod restarts. K3s ships with a local-path storage provisioner that makes this straightforward:
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: keycloak-data
namespace: keycloak
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
Then mount it in `keycloak-deployment.yaml` under the container spec:
```yaml
volumeMounts:
- name: keycloak-data
mountPath: /opt/keycloak/data
volumes:
- name: keycloak-data
persistentVolumeClaim:
claimName: keycloak-data
```
**Option 2 — Accept the behaviour and keep notes**
For a lab environment, losing configuration on restart is acceptable as long as you know how to recreate it quickly. Keeping your realm setup documented — which this troubleshooting article now serves as — means recovery takes a few minutes rather than a debugging session.
---
## Key Takeaways
- `CrashLoopBackOff` means the container is crashing on startup — always read logs with `--previous` to see why
- oauth2-proxy performs OIDC discovery at startup and will fail immediately if the Keycloak realm does not exist
- Keycloak in `start-dev` mode uses an ephemeral H2 database — all realm config is lost on pod restart
- The fix is always to recreate the realm, client, and users in Keycloak, then update oauth2-proxy with the new client secret
---
*Environment: K3s single-node on Ubuntu Server 26.04 LTS · Keycloak 26.3.3 · oauth2-proxy*