Writing · · 10 min read

No more static secrets

A leaked database password, a weekend of rotating it, and how we set things up so pods log in to Vault with their own service account instead.

Ivan RoussevDevOps & Platform Engineer
VaultKubernetesSecurity
On this page
  1. The idea
  2. Running Vault
  3. Letting Vault trust Kubernetes
  4. Roles and policies
  5. Getting secrets into the pod
  6. Database passwords that expire on their own
  7. What broke during the rollout
  8. Who can read the secrets now
  9. If I was starting on Monday

On a Friday afternoon a secret scanner flagged a commit in one of our Helm chart repos. Someone had copied a values-prod.yaml to debug something locally and pushed it to a branch. Inside it was the production Postgres password for the payments service, in plain text.

The repo was private and we have no sign anyone used it. Doesn't matter. Once a password is in Git history you treat it as leaked, so we had to rotate it.

That took the whole weekend.

The password was in a Kubernetes Secret in three clusters. It was in a CI variable for the migration job. It was in a CronJob that someone had set up a year earlier with its own copy. And one service we didn't even know about read the same Secret through an envFrom, so after we rotated it, that service went into CrashLoopBackOff and nobody knew why for two hours.

  1. Scanner flags the prod password in Git
  2. Rotating it in 3 clusters, CI and a CronJob
  3. A service nobody knew about crashloops
  4. Found it. Everything green again

Nothing about that was unusual. Most teams I've talked to keep secrets the same way: a base64 Secret (which is encoding, not encryption), created once by hand or by CI, never rotated because rotating is scary.

So we moved everything to Vault. This post is how we set it up, what we got wrong, and what I'd do differently.

The idea#

Putting secrets in Vault is the easy part. The hard part is how the pod logs in to Vault. If you give it a Vault token in a Kubernetes Secret, you're back where you started with one more system to run.

The fix is to not give pods credentials at all. Every pod already has an identity: its service account. Kubernetes gives each pod a signed, short-lived token for that service account. Vault can check that token with the Kubernetes API and decide what that pod is allowed to read.

  1. The pod sends its service account token to Vault and says which role it wants.
  2. Vault asks the Kubernetes API (TokenReview) if the token is real and still valid. This also catches tokens from pods that have been deleted.
  3. If the service account and namespace match the role, Vault gives back a short-lived Vault token with the role's policies.
  4. The pod uses that token to read its secrets, or to get database credentials that Vault creates right then.

No long-lived password sits in the cluster. If you delete the pod, its access is gone too.

Running Vault#

If Vault is down, new pods can't start. So we treated it like the most important thing in the platform from day one.

We run a dedicated Vault cluster and each Kubernetes cluster talks to it over the network. Each cluster gets its own login path in Vault, so if one cluster is compromised we can turn off trust for that cluster without touching the others. Running Vault inside the cluster it protects also gets weird fast (what happens when that cluster is the one that's broken?).

The setup:

  • Raft storage, three nodes, one per availability zone. No Consul to run.
  • Auto-unseal with AWS KMS. Unsealing Vault by hand works until a node restarts at 3am. The Vault pods get access to the KMS key through IRSA, not a static AWS key, otherwise you just moved the problem again.
  • TLS on everything.

The part of values.yaml for the official Helm chart that matters:

values.yaml
server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file   = "/vault/userconfig/vault-tls/tls.crt"
          tls_key_file    = "/vault/userconfig/vault-tls/tls.key"
        }

        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr     = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
          retry_join {
            leader_api_addr     = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
          retry_join {
            leader_api_addr     = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
        }

        seal "awskms" {
          region     = "eu-central-1"
          kms_key_id = "alias/vault-prod-unseal"
        }

        service_registration "kubernetes" {}

Plus pod anti-affinity on topology.kubernetes.io/zone and a PodDisruptionBudget, so a node drain never takes out two Vault pods at once.

One thing to know: Vault has been under the Business Source License since 2023. If that's a problem for you, OpenBao is the open source fork and everything here works the same with bao instead of vault.

Letting Vault trust Kubernetes#

Vault has two ways to accept a service account token, and this is the one decision I'd spend real time on.

Kubernetes authJWT / OIDC auth
How it checks the tokenCalls the cluster's TokenReview APIChecks the signature itself using the cluster's public keys
Notices a deleted podYesNo, the token works until it expires
Vault needs to reach the API serverYesNo

We went with Kubernetes auth. Our Vault could reach every API server, and I liked that a token stolen from a dead pod is useless. If your Vault can't reach the API servers, use JWT auth and keep token lifetimes short. We never ran it in production, so I won't pretend to know where it bites.

For the TokenReview call, Vault needs permission to ask the API. Since our Vault runs outside the clusters, we let Vault use the token the pod sends as the reviewer. No extra long-lived token to store and rotate. The catch is that every service account that logs in needs the system:auth-delegator role, so we bind it per namespace with a group:

vault-auth-delegator.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-client-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: Group
    name: system:serviceaccounts:payments
    apiGroup: rbac.authorization.k8s.io
  - kind: Group
    name: system:serviceaccounts:orders
    apiGroup: rbac.authorization.k8s.io

Then one auth mount per cluster, named after the cluster, so the audit log tells you where a login came from:

terminal
vault auth enable -path=k8s-prod-eu1 kubernetes

vault write auth/k8s-prod-eu1/config \
    kubernetes_host="https://<api-server-endpoint>:443" \
    kubernetes_ca_cert=@cluster-ca.crt \
    disable_local_ca_jwt=true

Roles and policies#

A role says: these service accounts, in these namespaces, get these policies, for this long.

terminal
vault write auth/k8s-prod-eu1/role/payments-api \
    bound_service_account_names="payments-api" \
    bound_service_account_namespaces="payments" \
    audience="vault" \
    alias_name_source="serviceaccount_name" \
    token_policies="app-kv-read,payments-db" \
    token_ttl=1h \
    token_max_ttl=4h \
    token_no_default_policy=true

A few things I'd push back on in a code review:

  • Never * for both names and namespaces. That means any pod in the cluster can log in. I've seen it in "temporary" configs more than once.
  • Set audience="vault". The default pod token is meant for the Kubernetes API. A token meant for one thing shouldn't work as a login somewhere else, so pods get a separate token made for Vault (you'll see it in the deployment below).
  • alias_name_source="serviceaccount_name" keeps the same Vault identity when Argo CD deletes and recreates a service account. Good with GitOps, but it also means whoever can create a service account with that name gets the same access. More on that later.

For policies, we started with one per app. That was fine at 10 apps and annoying at 40. Vault can fill in the namespace from the login, so one policy covers everyone:

app-kv-read.hcl
# every workload reads secrets under its own namespace, nothing else
path "secret/data/{{identity.entity.aliases.auth_kubernetes_1a2b3c4d.metadata.service_account_namespace}}/*" {
  capabilities = ["read"]
}

path "secret/metadata/{{identity.entity.aliases.auth_kubernetes_1a2b3c4d.metadata.service_account_namespace}}/*" {
  capabilities = ["list"]
}

The auth_kubernetes_... part is your mount's accessor (vault auth list shows it). Now payments can read secret/payments/* and can't read secret/orders/*. A new team just writes their secrets under their namespace's path and doesn't need a new policy.

All of this lives in Terraform with the Vault provider and goes through pull requests. A policy change is a permissions change and should get reviewed like one.

Getting secrets into the pod#

We use two options and that's it:

Where the secret ends upWhen we use it
Vault Secrets Operator (VSO)A normal Kubernetes SecretMost apps, and any Helm chart we don't own that expects a Secret
Vault Agent InjectorA file in memory inside the podPayment provider keys, signing keys, anything we don't want in etcd at all

There's also the Secrets Store CSI driver and using the Vault SDK in the app. Both are fine, we just didn't need them.

VSO syncs Vault into a normal Secret, so nothing changes for the app. The secret is in etcd again, so turn on encryption at rest for Secrets first. The part that made the switch worth it for us is rolloutRestartTargets: change the value in Vault and VSO updates the Secret and restarts the deployment. That's the exact thing that cost us a weekend.

payments-api-vault.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: payments-api
  namespace: payments
spec:
  method: kubernetes
  mount: k8s-prod-eu1
  kubernetes:
    role: payments-api
    serviceAccount: payments-api
    audiences: ["vault"]
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: payments-api-config
  namespace: payments
spec:
  vaultAuthRef: payments-api
  type: kv-v2
  mount: secret
  path: payments/api
  refreshAfter: 60s
  destination:
    name: payments-api-config
    create: true
  rolloutRestartTargets:
    - kind: Deployment
      name: payments-api

The Agent Injector adds a sidecar that logs in to Vault, writes the secrets to a file in memory and keeps them fresh. The secret never becomes a Kubernetes object:

deployment.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/auth-path: "auth/k8s-prod-eu1"
        vault.hashicorp.com/role: "payments-api"
        # log in with the Vault token below, not the default one
        vault.hashicorp.com/agent-service-account-token-volume-name: "vault-token"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/payments/api"
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/payments/api" -}}
          STRIPE_KEY={{ .Data.data.stripe_key }}
          {{- end }}
    spec:
      serviceAccountName: payments-api
      automountServiceAccountToken: false
      volumes:
        - name: vault-token
          projected:
            sources:
              - serviceAccountToken:
                  path: token
                  audience: vault
                  expirationSeconds: 600

Database passwords that expire on their own#

This is the part that actually fixes the Friday problem. With Vault's database secrets engine, every pod gets its own Postgres user that lives for an hour. There's no shared password to leak.

terminal
vault secrets enable database

vault write database/config/payments-pg \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@payments-pg.payments.svc:5432/payments?sslmode=verify-full" \
    username="vault-admin" \
    password="<bootstrap-password>" \
    allowed_roles="payments-api"

# rotate the bootstrap password right away so only Vault knows it
vault write -force database/rotate-root/payments-pg

vault write database/roles/payments-api \
    db_name=payments-pg \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
                         GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl=1h \
    max_ttl=24h

If that leaked commit had happened after this, the password in it would have stopped working within the hour. And if we ever need to cut access right now, it's one command:

terminal
vault lease revoke -prefix database/creds/payments-api

What broke during the rollout#

It didn't go smoothly.

The one-hour cliff

Incident 01
Symptom
The first service we moved to dynamic DB credentials threw auth errors exactly 60 minutes after every deploy.
Cause
Vault was doing its job and the credentials expired. The app's connection pool opened connections at startup and never picked up the new username and password.
Fix
The app re-reads the credentials file and rebuilds the pool when it changes. Test this before you roll it out, not after.

Audit logs took down logins

Incident 02
Symptom
One morning every new pod in every cluster failed to start, because none of them could log in to Vault.
Cause
Vault refuses requests if it can't write them to at least one audit device. We had one: a file on a volume we sized way too small, and it filled up.
Fix
Two audit devices (a file plus a socket to our log pipeline) and an alert on disk usage.

Vault down means deploys stop

Incident 03
Symptom
During a short Vault outage, running pods were fine but nothing new could start. No deploys, no autoscaling, no node replacements.
Cause
Every new pod has to log in to Vault before it gets its secrets.
Fix
Vault is now on the same tier as the API server for alerting, on-call and upgrades.

Who can read the secrets now#

This is the part people skip. With this setup, "who can read the payments secrets?" becomes "who can run a pod as the payments-api service account?". That's anyone who can create Deployments, Jobs or CronJobs in the payments namespace.

So your Kubernetes RBAC is now your secrets permissions. We tightened a few things:

  • No more cluster-wide edit for teams. Write access is per namespace.
  • An admission policy so only Argo CD can create service accounts, and pods can only use service accounts from an allowed list. We use Kyverno for this.
  • automountServiceAccountToken: false on pods that don't talk to the API, and the Vault token only where it's needed.

It doesn't fix everything. An app that reads a secret can still log it or send it to an error tracker. Short-lived credentials make that hurt less, they don't stop it.

If I was starting on Monday#

  1. Set up Vault with Raft, KMS auto-unseal, TLS, two audit devices and automatic snapshots. Then break it on purpose and restore a snapshot before anything depends on it.
  2. Set up Kubernetes auth and the namespace policy in Terraform.
  3. Move one boring, non-critical service to VSO. Nothing should change for the app.
  4. Rotate every secret you move. The old copies in Git and CI should be useless.
  5. Roll out namespace by namespace, then move databases to dynamic credentials, starting with the one that would hurt most if it leaked.

Things I'm still not sure about: whether VSO putting secrets back into etcd is OK for everything, or whether we'll end up moving more apps to the injector. And whether JWT auth makes more sense as we add clusters Vault can't easily reach. If you've run either at scale, I'd like to hear how it went.