On this page
On a Friday afternoon a secret scanner flagged a commit in one of our Helm chart repos. Someone had copied a values-prod.yaml to debug something locally and pushed it to a branch. Inside it was the production Postgres password for the payments service, in plain text.
The repo was private and we have no sign anyone used it. Doesn't matter. Once a password is in Git history you treat it as leaked, so we had to rotate it.
That took the whole weekend.
The password was in a Kubernetes Secret in three clusters. It was in a CI variable for the migration job. It was in a CronJob that someone had set up a year earlier with its own copy. And one service we didn't even know about read the same Secret through an envFrom, so after we rotated it, that service went into CrashLoopBackOff and nobody knew why for two hours.
- Scanner flags the prod password in Git
- Rotating it in 3 clusters, CI and a CronJob
- A service nobody knew about crashloops
- Found it. Everything green again
Nothing about that was unusual. Most teams I've talked to keep secrets the same way: a base64 Secret (which is encoding, not encryption), created once by hand or by CI, never rotated because rotating is scary.
So we moved everything to Vault. This post is how we set it up, what we got wrong, and what I'd do differently.
The idea#
Putting secrets in Vault is the easy part. The hard part is how the pod logs in to Vault. If you give it a Vault token in a Kubernetes Secret, you're back where you started with one more system to run.
The fix is to not give pods credentials at all. Every pod already has an identity: its service account. Kubernetes gives each pod a signed, short-lived token for that service account. Vault can check that token with the Kubernetes API and decide what that pod is allowed to read.
- The pod sends its service account token to Vault and says which role it wants.
- Vault asks the Kubernetes API (
TokenReview) if the token is real and still valid. This also catches tokens from pods that have been deleted. - If the service account and namespace match the role, Vault gives back a short-lived Vault token with the role's policies.
- The pod uses that token to read its secrets, or to get database credentials that Vault creates right then.
No long-lived password sits in the cluster. If you delete the pod, its access is gone too.
Running Vault#
If Vault is down, new pods can't start. So we treated it like the most important thing in the platform from day one.
We run a dedicated Vault cluster and each Kubernetes cluster talks to it over the network. Each cluster gets its own login path in Vault, so if one cluster is compromised we can turn off trust for that cluster without touching the others. Running Vault inside the cluster it protects also gets weird fast (what happens when that cluster is the one that's broken?).
The setup:
- Raft storage, three nodes, one per availability zone. No Consul to run.
- Auto-unseal with AWS KMS. Unsealing Vault by hand works until a node restarts at 3am. The Vault pods get access to the KMS key through IRSA, not a static AWS key, otherwise you just moved the problem again.
- TLS on everything.
The part of values.yaml for the official Helm chart that matters:
server:
ha:
enabled: true
replicas: 3
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-tls/tls.crt"
tls_key_file = "/vault/userconfig/vault-tls/tls.key"
}
storage "raft" {
path = "/vault/data"
retry_join {
leader_api_addr = "https://vault-0.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://vault-1.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://vault-2.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
}
}
seal "awskms" {
region = "eu-central-1"
kms_key_id = "alias/vault-prod-unseal"
}
service_registration "kubernetes" {}Plus pod anti-affinity on topology.kubernetes.io/zone and a PodDisruptionBudget, so a node drain never takes out two Vault pods at once.
One thing to know: Vault has been under the Business Source License since 2023. If that's a problem for you, OpenBao is the open source fork and everything here works the same with bao instead of vault.
Letting Vault trust Kubernetes#
Vault has two ways to accept a service account token, and this is the one decision I'd spend real time on.
| Kubernetes auth | JWT / OIDC auth | |
|---|---|---|
| How it checks the token | Calls the cluster's TokenReview API | Checks the signature itself using the cluster's public keys |
| Notices a deleted pod | Yes | No, the token works until it expires |
| Vault needs to reach the API server | Yes | No |
We went with Kubernetes auth. Our Vault could reach every API server, and I liked that a token stolen from a dead pod is useless. If your Vault can't reach the API servers, use JWT auth and keep token lifetimes short. We never ran it in production, so I won't pretend to know where it bites.
For the TokenReview call, Vault needs permission to ask the API. Since our Vault runs outside the clusters, we let Vault use the token the pod sends as the reviewer. No extra long-lived token to store and rotate. The catch is that every service account that logs in needs the system:auth-delegator role, so we bind it per namespace with a group:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-client-auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: Group
name: system:serviceaccounts:payments
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts:orders
apiGroup: rbac.authorization.k8s.ioThen one auth mount per cluster, named after the cluster, so the audit log tells you where a login came from:
vault auth enable -path=k8s-prod-eu1 kubernetes
vault write auth/k8s-prod-eu1/config \
kubernetes_host="https://<api-server-endpoint>:443" \
kubernetes_ca_cert=@cluster-ca.crt \
disable_local_ca_jwt=trueRoles and policies#
A role says: these service accounts, in these namespaces, get these policies, for this long.
vault write auth/k8s-prod-eu1/role/payments-api \
bound_service_account_names="payments-api" \
bound_service_account_namespaces="payments" \
audience="vault" \
alias_name_source="serviceaccount_name" \
token_policies="app-kv-read,payments-db" \
token_ttl=1h \
token_max_ttl=4h \
token_no_default_policy=trueA few things I'd push back on in a code review:
- Never
*for both names and namespaces. That means any pod in the cluster can log in. I've seen it in "temporary" configs more than once. - Set
audience="vault". The default pod token is meant for the Kubernetes API. A token meant for one thing shouldn't work as a login somewhere else, so pods get a separate token made for Vault (you'll see it in the deployment below). alias_name_source="serviceaccount_name"keeps the same Vault identity when Argo CD deletes and recreates a service account. Good with GitOps, but it also means whoever can create a service account with that name gets the same access. More on that later.
For policies, we started with one per app. That was fine at 10 apps and annoying at 40. Vault can fill in the namespace from the login, so one policy covers everyone:
# every workload reads secrets under its own namespace, nothing else
path "secret/data/{{identity.entity.aliases.auth_kubernetes_1a2b3c4d.metadata.service_account_namespace}}/*" {
capabilities = ["read"]
}
path "secret/metadata/{{identity.entity.aliases.auth_kubernetes_1a2b3c4d.metadata.service_account_namespace}}/*" {
capabilities = ["list"]
}The auth_kubernetes_... part is your mount's accessor (vault auth list shows it). Now payments can read secret/payments/* and can't read secret/orders/*. A new team just writes their secrets under their namespace's path and doesn't need a new policy.
All of this lives in Terraform with the Vault provider and goes through pull requests. A policy change is a permissions change and should get reviewed like one.
Getting secrets into the pod#
We use two options and that's it:
| Where the secret ends up | When we use it | |
|---|---|---|
| Vault Secrets Operator (VSO) | A normal Kubernetes Secret | Most apps, and any Helm chart we don't own that expects a Secret |
| Vault Agent Injector | A file in memory inside the pod | Payment provider keys, signing keys, anything we don't want in etcd at all |
There's also the Secrets Store CSI driver and using the Vault SDK in the app. Both are fine, we just didn't need them.
VSO syncs Vault into a normal Secret, so nothing changes for the app. The secret is in etcd again, so turn on encryption at rest for Secrets first. The part that made the switch worth it for us is rolloutRestartTargets: change the value in Vault and VSO updates the Secret and restarts the deployment. That's the exact thing that cost us a weekend.
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: payments-api
namespace: payments
spec:
method: kubernetes
mount: k8s-prod-eu1
kubernetes:
role: payments-api
serviceAccount: payments-api
audiences: ["vault"]
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: payments-api-config
namespace: payments
spec:
vaultAuthRef: payments-api
type: kv-v2
mount: secret
path: payments/api
refreshAfter: 60s
destination:
name: payments-api-config
create: true
rolloutRestartTargets:
- kind: Deployment
name: payments-apiThe Agent Injector adds a sidecar that logs in to Vault, writes the secrets to a file in memory and keeps them fresh. The secret never becomes a Kubernetes object:
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/auth-path: "auth/k8s-prod-eu1"
vault.hashicorp.com/role: "payments-api"
# log in with the Vault token below, not the default one
vault.hashicorp.com/agent-service-account-token-volume-name: "vault-token"
vault.hashicorp.com/agent-inject-secret-config: "secret/data/payments/api"
vault.hashicorp.com/agent-inject-template-config: |
{{- with secret "secret/data/payments/api" -}}
STRIPE_KEY={{ .Data.data.stripe_key }}
{{- end }}
spec:
serviceAccountName: payments-api
automountServiceAccountToken: false
volumes:
- name: vault-token
projected:
sources:
- serviceAccountToken:
path: token
audience: vault
expirationSeconds: 600Database passwords that expire on their own#
This is the part that actually fixes the Friday problem. With Vault's database secrets engine, every pod gets its own Postgres user that lives for an hour. There's no shared password to leak.
vault secrets enable database
vault write database/config/payments-pg \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@payments-pg.payments.svc:5432/payments?sslmode=verify-full" \
username="vault-admin" \
password="<bootstrap-password>" \
allowed_roles="payments-api"
# rotate the bootstrap password right away so only Vault knows it
vault write -force database/rotate-root/payments-pg
vault write database/roles/payments-api \
db_name=payments-pg \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl=1h \
max_ttl=24hIf that leaked commit had happened after this, the password in it would have stopped working within the hour. And if we ever need to cut access right now, it's one command:
vault lease revoke -prefix database/creds/payments-apiWhat broke during the rollout#
It didn't go smoothly.
The one-hour cliff
Incident 01- Symptom
- The first service we moved to dynamic DB credentials threw auth errors exactly 60 minutes after every deploy.
- Cause
- Vault was doing its job and the credentials expired. The app's connection pool opened connections at startup and never picked up the new username and password.
- Fix
- The app re-reads the credentials file and rebuilds the pool when it changes. Test this before you roll it out, not after.
Audit logs took down logins
Incident 02- Symptom
- One morning every new pod in every cluster failed to start, because none of them could log in to Vault.
- Cause
- Vault refuses requests if it can't write them to at least one audit device. We had one: a file on a volume we sized way too small, and it filled up.
- Fix
- Two audit devices (a file plus a socket to our log pipeline) and an alert on disk usage.
Vault down means deploys stop
Incident 03- Symptom
- During a short Vault outage, running pods were fine but nothing new could start. No deploys, no autoscaling, no node replacements.
- Cause
- Every new pod has to log in to Vault before it gets its secrets.
- Fix
- Vault is now on the same tier as the API server for alerting, on-call and upgrades.
Who can read the secrets now#
This is the part people skip. With this setup, "who can read the payments secrets?" becomes "who can run a pod as the payments-api service account?". That's anyone who can create Deployments, Jobs or CronJobs in the payments namespace.
So your Kubernetes RBAC is now your secrets permissions. We tightened a few things:
- No more cluster-wide
editfor teams. Write access is per namespace. - An admission policy so only Argo CD can create service accounts, and pods can only use service accounts from an allowed list. We use Kyverno for this.
automountServiceAccountToken: falseon pods that don't talk to the API, and the Vault token only where it's needed.
It doesn't fix everything. An app that reads a secret can still log it or send it to an error tracker. Short-lived credentials make that hurt less, they don't stop it.
If I was starting on Monday#
- Set up Vault with Raft, KMS auto-unseal, TLS, two audit devices and automatic snapshots. Then break it on purpose and restore a snapshot before anything depends on it.
- Set up Kubernetes auth and the namespace policy in Terraform.
- Move one boring, non-critical service to VSO. Nothing should change for the app.
- Rotate every secret you move. The old copies in Git and CI should be useless.
- Roll out namespace by namespace, then move databases to dynamic credentials, starting with the one that would hurt most if it leaked.
Things I'm still not sure about: whether VSO putting secrets back into etcd is OK for everything, or whether we'll end up moving more apps to the injector. And whether JWT auth makes more sense as we add clusters Vault can't easily reach. If you've run either at scale, I'd like to hear how it went.