OnCallReadyStart learning

Security and secrets Lesson 24.27 · Vault & Secrets Management · 36 min read

Vault on Kubernetes: the auth method, Agent Injector, ESO and CSI

In plain words

Think of a big office building where every employee wears a badge the building itself prints. The safe room on the top floor does not keep a list of passwords: when someone knocks, the guard phones the badge office and asks "is this badge real, and whose is it?". If the badge belongs to someone on the safe room's list, the door opens for an hour.

In Kubernetes the badge is the pod's ServiceAccount token, the phone call is a TokenReview to the API server, and the list is a Vault role that maps a ServiceAccount in a namespace to policies. The Agent Injector is a helper who stands next to the app and collects the papers for it, writing them into files; External Secrets Operator copies them into a normal Kubernetes Secret instead.

The problem: a pod needs a password

The orders API runs in Kubernetes and needs its database password. You have already seen the two bad answers: the password in the image (anyone who pulls the image has it) and a Kubernetes Secret written in git (base64 is an encoding, not encryption - anyone who can read the repo can decode it). The password belongs in Vault. The question this lesson answers is: how does the pod get it out, without a Vault token baked into the pod either?

What you need to know already: ServiceAccounts and their tokens (19.33), RBAC and ClusterRoleBindings (19.30), Secrets and why base64 is not security (19.41), init containers and sidecars (17.35), the apiserver's request path and admission (17.5), CRDs (18.26), and this chapter's lessons on the Kubernetes-free basics: tokens and auth methods, KV v2 and the data/ path, policies, and Vault Agent (auto-auth, sinks, templates).

The words you need first

  • Workload identity - proof of "which program is this" that the platform hands out, so the program never needs a stored password to prove it. In Kubernetes it is the pod's ServiceAccount token: a signed JWT the kubelet mounts at /var/run/secrets/kubernetes.io/serviceaccount/token.
  • TokenReview - the Kubernetes API call that answers "is this JWT valid, and whose is it?". Anyone who wants to trust a ServiceAccount token asks the apiserver with a TokenReview.
  • Mutating webhook - an admission step (17.5) that may change an object on its way into the cluster. The Vault Agent Injector is one: it rewrites pods.
  • Secret zero - the first credential a program needs to fetch all the others. The whole game is making secret zero something the platform gives you (a ServiceAccount token) instead of something you store.

Four ways to get a secret into a pod

howthe pod seesgood for
the app calls Vaultan SDK in the app logs in with its ServiceAccount tokenwhatever the code readsapps you own that need dynamic secrets
Agent Injectora mutating webhook adds Vault Agent as an init container + sidecarfiles in /vault/secrets/file-based config, dynamic secrets, no app changes
External Secrets Operatora controller copies Vault values into a Kubernetes Secreta normal Secret (env or volume)manifests and charts that expect Secrets, many providers
Secrets Store CSI drivera CSI volume plugin mounts secrets at pod startfiles in a volumefile mounts without a sidecar

All four start the same way: the pod (or the controller acting for it) logs in to Vault with a Kubernetes ServiceAccount token through Vault's kubernetes auth method. So that comes first.

The Kubernetes auth method

The flow, every time a pod logs in:

  1. The pod reads its ServiceAccount JWT and sends it to Vault: PUT /v1/auth/kubernetes/login with role=orders jwt=....
  2. Vault does not trust the JWT on its own: it sends a TokenReview to the cluster's apiserver ("is this token valid?").
  3. The apiserver answers with the token's user: system:serviceaccount:shop:orders.
  4. Vault checks the answer against the role: is the ServiceAccount name in bound_service_account_names, the namespace in bound_service_account_namespaces?
  5. If yes, Vault issues a Vault token with the role's token_policies and TTL.

Setting it up is three writes. Vault here runs on the box (oncall-lab, 10.64.0.2), outside the cluster, and talks to the apiserver at https://10.64.0.10:6443:

# enabling it (the lab's setup has done it - an illustration, no ▶)
$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
$ vault write auth/kubernetes/config kubernetes_host=https://10.64.0.10:6443
Success! Data written to: auth/kubernetes/config
$ vault write auth/kubernetes/role/orders bound_service_account_names=orders bound_service_account_namespaces=shop token_policies=orders-db ttl=1h
Success! Data written to: auth/kubernetes/role/orders

vault read shows what the role really holds - note that ttl is the old name of token_ttl, and both are shown:

$ vault read auth/kubernetes/role/orders
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
audience                                    n/a
bound_service_account_names                 [orders]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [shop]
max_ttl                                     0s
period                                      0s
policies                                    [orders-db]
token_bound_cidrs                           []
token_explicit_max_ttl                      0s
token_max_ttl                               0s
token_no_default_policy                     false
token_num_uses                              0
token_period                                0s
token_policies                              [orders-db]
token_ttl                                   1h
token_type                                  default
ttl                                         1h

You can play the pod by hand: kubectl create token asks the apiserver for a fresh ServiceAccount token (19.33), and vault write auth/kubernetes/login is exactly the call the agent makes.

Who is allowed to ask "is this token valid?"

A TokenReview is itself an API call, and RBAC (19.30) guards it: the caller needs create on tokenreviews.authentication.k8s.io. The built-in ClusterRole system:auth-delegator grants exactly that. Two ways to wire it:

  • A reviewer ServiceAccount (the usual production setup): one ServiceAccount (say vault/vault-auth) gets system:auth-delegator, and its long-lived token goes into auth/kubernetes/config token_reviewer_jwt=.... Vault uses it for every review.
  • The client's own JWT (what you get with no token_reviewer_jwt and Vault outside the cluster): every application ServiceAccount must be bound to system:auth-delegator itself. Simple for a lab, noisy for a platform.

When Vault runs inside the cluster (the official chart), it can use its own pod's token as the reviewer and you configure nothing.

$ kubectl create clusterrolebinding vault-tokenreview --clusterrole=system:auth-delegator --serviceaccount=shop:orders
clusterrolebinding.rbac.authorization.k8s.io/vault-tokenreview created
$ vault write auth/kubernetes/login role=orders jwt=$(kubectl create token orders -n shop)
Key                                       Value
---                                       -----
token                                     hvs.CAESIK7YFJ0lWMnHhAbkwrod9uxbryuY18vYsaposwGh4KHGh2cy4q8AdXLB0SKv8C1qmhudL59Ik0YDUR8ot
token_accessor                            Hc74jlxxLNQxeUUHnOErZkPQ
token_duration                            1h
token_renewable                           true
token_policies                            ["default" "orders-db"]
identity_policies                         []
policies                                  ["default" "orders-db"]
token_meta_role                           orders
token_meta_service_account_name           orders
token_meta_service_account_namespace      shop
token_meta_service_account_secret_name    n/a
token_meta_service_account_uid            4dc32e95-13bc-4346-96a0-0650bd54c0c5

The token's metadata records which ServiceAccount it was issued to: that is what the audit log shows when you later ask "who read this secret?".

The error ladder

Read the error top to bottom; each one means the previous step passed:

errormeaningfix
invalid role name "x"no such role on that mountvault list auth/kubernetes/role
permission denied (403)the TokenReview failed: the reviewer may not create tokenreviews, or the JWT is invalid / expired / for a deleted ServiceAccountsystem:auth-delegator, a fresh token
service account name not authorizedthe review passed, the name is not in the rolebound_service_account_names
namespace not authorizedthe name matched, the namespace did notbound_service_account_namespaces

The Agent Injector

The injector is a small webhook server (hashicorp/vault-k8s, 1.7.6 at the time of writing), normally installed with the official Vault chart (injector.enabled=true, and injector.externalVaultAddr when Vault lives outside the cluster, as here). In the lab the platform team has already installed it:

$ kubectl get mutatingwebhookconfigurations
NAME                       AGE
vault-agent-injector-cfg   15m
$ kubectl -n vault get deploy,pods
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           15m

NAME                                        READY   STATUS    RESTARTS   AGE
pod/vault-agent-injector-z8xd88smw9-8zfgw   1/1     Running   0          15m

You never talk to it. You annotate the pod template, and every new pod that carries vault.hashicorp.com/agent-inject: "true" is rewritten on its way in:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "orders"
        vault.hashicorp.com/agent-inject-secret-db: "secret/data/shop/orders/db"
    spec:
      serviceAccountName: orders
  • agent-inject: "true" - inject. A quoted string: YAML true without quotes is a boolean, and annotations must be strings.
  • role - the Vault Kubernetes auth role to log in with.
  • agent-inject-secret-<name>: <path> - render the secret at that API path (for KV v2 that is secret/data/..., not the vault kv get path) into the file /vault/secrets/<name>.

What the webhook adds, in short:

  • an init container vault-agent-init: logs in, renders every file once, exits. Your app container only starts after it, so the file is there on the app's first read;
  • a sidecar vault-agent: stays running, renews the token, re-renders when a secret changes or a lease nears its end;
  • a shared in-memory emptyDir volume mounted at /vault/secrets in every container.
# from the injector lab, once the annotated Deployment is applied (an illustration, no ▶ - you build it in the next mission)
$ kubectl get pods -n shop
NAME                          READY   STATUS     RESTARTS   AGE
orders-api-ss6zfq5cj5-r9lj2   0/2     Init:0/1   0          3s
$ kubectl get pods -n shop
NAME                          READY   STATUS    RESTARTS   AGE
orders-api-ss6zfq5cj5-r9lj2   2/2     Running   0          33s

2/2: your container and the sidecar. A pod stuck at Init:0/1 means the init agent cannot log in or cannot read a secret - its log says which: kubectl logs <pod> -c vault-agent-init. A pod that comes up 1/1 with no vault-agent container means the injector never saw it: no webhook, the annotation on the Deployment's metadata instead of the pod template, or a typo.

# from the injector lab, once the annotated Deployment is applied (an illustration, no ▶ - you build it in the next mission)
$ kubectl logs -n shop deploy/orders-api -c vault-agent | tail -5
2026-09-22T20:00:50.000Z [INFO]  agent.template.server: template server received new token
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) creating watcher
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) starting
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/db"

The default template is a Go map dump

Without a template annotation, the injector renders the whole secret with Go's default formatting. For KV v2 that is rarely what an app can read:

# from the injector lab, once the annotated Deployment is applied (an illustration, no ▶ - you build it in the next mission)
$ kubectl exec -n shop deploy/orders-api -c app -- cat /vault/secrets/db
data: map[DB_PASSWORD:orders-static-2023 DB_USER:orders_app]
metadata: map[created_time:2026-09-22T20:00:38.104545576Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Give the file a shape with agent-inject-template-<name> - the same template language as Vault Agent (consul-template):

        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "secret/data/shop/orders/db" -}}
          DB_USER={{ .Data.data.DB_USER }}
          DB_PASSWORD={{ .Data.data.DB_PASSWORD }}
          {{- end }}

.Data is the API response's data; for KV v2 the values sit one level deeper, in .Data.data. For a dynamic database secret (no extra level) it is .Data.username. Other annotations you will meet: agent-pre-populate-only: "true" (only the init container, no sidecar: for Jobs, or when rotation needs a restart anyway), agent-inject-command-<name> (run a command after a render, e.g. send the app a SIGHUP), agent-inject-file-<name> (another file name), service (another Vault address).

External Secrets Operator

The injector puts files in pods. Many teams instead want a plain Kubernetes Secret: their charts and manifests expect one, deployment tools compare objects, and other operators read Secrets. External Secrets Operator (ESO) is a controller that keeps a Secret in sync with an external store (Vault, cloud secret managers and many more). The Vault side is split in two objects:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore          # platform-owned: HOW to reach Vault
metadata:
  name: vault
spec:
  provider:
    vault:
      server: https://10.64.0.2:8200
      path: secret                # the KV mount
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret              # team-owned: WHAT to copy, into which Secret
metadata:
  name: payments-api-key
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault
  target:
    name: payments-api-key
  data:
  - secretKey: API_KEY
    remoteRef:
      key: shop/payments/api-key  # the key inside the mount: NO data/ here
      property: API_KEY

Two path conventions in one chapter: the injector wants the full API path (secret/data/shop/payments/api-key), the ESO vault provider wants the mount in path and the key without data/ (it adds it because version: v2).

The store logs in as the operator's ServiceAccount, so the same error ladder applies, shown in the store's status:

# from the External Secrets lab (an illustration, no ▶ - you build it in the mission after next)
$ kubectl get clustersecretstore vault
NAME    AGE   STATUS                  CAPABILITIES   READY
vault   5s    InvalidProviderConfig   ReadWrite      False
$ kubectl get clustersecretstore vault -o jsonpath='{.status.conditions[0].message}{"\n"}'
unable to log in to auth method: unable to log in with Kubernetes auth: Error making API request.

URL: PUT https://10.64.0.2:8200/v1/auth/kubernetes/login
Code: 400. Errors:

* invalid role name "eso"

With the role and the auth-delegator binding in place:

# from the External Secrets lab (an illustration, no ▶ - you build it in the mission after next)
$ kubectl get clustersecretstore vault
NAME    AGE   STATUS   CAPABILITIES   READY
vault   76s   Valid    ReadWrite      True
$ kubectl get externalsecret -n shop
NAME               STORETYPE            STORE   REFRESH INTERVAL   STATUS         READY
payments-api-key   ClusterSecretStore   vault   1m                 SecretSynced   True
$ kubectl get secret payments-api-key -n shop -o jsonpath='{.data.API_KEY}' | base64 -d; echo
pk_live_7f3a9c21e8

A changed value in Vault reaches the Secret within refreshInterval. To skip the wait, change the force-sync annotation - ESO syncs on any change of it:

# from the External Secrets lab (an illustration, no ▶ - you build it in the mission after next)
$ kubectl annotate externalsecret payments-api-key -n shop force-sync=$(date +%s) --overwrite
externalsecret.external-secrets.io/payments-api-key annotated

A wrong key does not break the existing Secret: it keeps the last good value and the ExternalSecret turns SecretSyncedError, with the reason in its events:

# from the External Secrets lab (an illustration, no ▶ - you build it in the mission after next)
$ kubectl describe externalsecret payments-api-key -n shop | tail -1
  Warning  UpdateFailed  8s   external-secrets  error retrieving secret at .data[0], key: shop/payments/apikey, err: Secret does not exist

The price of ESO: the value is now in etcd as a Kubernetes Secret (19.41), so anyone with get secrets in the namespace reads it, and you need encryption at rest for etcd and tight RBAC. The injector never writes the secret to the Kubernetes API.

The CSI driver and the Vault Secrets Operator

Two more you will see in job ads:

  • Secrets Store CSI driver with the Vault provider: a SecretProviderClass object and a CSI volume in the pod. The kubelet mounts the secret as files at pod start; no sidecar. Optional sync to a Secret. Rotation by polling.
  • Vault Secrets Operator (VSO): HashiCorp's own operator, VaultStaticSecret and VaultDynamicSecret objects that write Kubernetes Secrets - like ESO but Vault-only, and it can restart a Deployment when a secret changes.

(simulator) The lab cluster has the injector and ESO; the CSI driver and VSO are described here, not installed.

Choosing

  • Charts and manifests that expect Secrets, several secret stores: ESO (or VSO).
  • No secret in the Kubernetes API, dynamic secrets that must be renewed, files: the injector.
  • Only files, no sidecar per pod: the CSI driver.
  • The app needs a fresh credential per request, or transit encryption: call Vault from the app with its ServiceAccount token.

In an interview: "the pod never holds a Vault token in its image or manifest: it proves who it is with its ServiceAccount token, Vault checks that with a TokenReview, and the role maps ServiceAccount + namespace to policies."

What you can now do

  • Configure the Kubernetes auth method and a role, and log in by hand with kubectl create token.
  • Read the login error ladder: invalid role name, permission denied (TokenReview / system:auth-delegator), service account name not authorized, namespace not authorized.
  • Annotate a pod template for the Agent Injector, template the file, and debug a pod stuck at Init:0/1.
  • Sync a Vault value into a Secret with ESO, read the store's and the ExternalSecret's status, and force a refresh.
  • Say when you would pick the injector, ESO, the CSI driver or the app itself.

Why it helps

Almost every platform team that runs Vault has to answer "how do pods get their secrets?", and the answer decides where secrets end up: only in files inside pods (the injector), also in etcd as Secrets (ESO), or inside the app (an SDK). On call, the failures are always the same few - pods stuck at Init:0/1, permission denied because the reviewer lacks system:auth-delegator, namespace not authorized, an ExternalSecret in SecretSyncedError - and you fix them in minutes once you can read the login error ladder. In design reviews and interviews, choosing between the injector, ESO and the CSI driver, and explaining "secret zero", is a standard question for platform roles.

Commands in this lesson

vault kubectl

FAQ

Why does Vault need to call the Kubernetes API at all?

Because a ServiceAccount token is only a signed JWT: Vault cannot tell on its own whether it is still valid, revoked, or belongs to a deleted ServiceAccount. It sends a TokenReview to the API server, which checks the signature and the ServiceAccount and answers with the identity. Only then does Vault compare the name and namespace with the role.

What is system:auth-delegator and who needs it?

A built-in ClusterRole that allows creating TokenReviews (and SubjectAccessReviews). Whoever performs the review needs it: a dedicated reviewer ServiceAccount whose token you put in token_reviewer_jwt, or - when Vault runs outside the cluster with no reviewer token - every application ServiceAccount, because Vault then reviews each token with that same token.

My pod is stuck at Init:0/1. Where do I look?

At the init container's log: kubectl logs <pod> -c vault-agent-init. It shows the agent's login and template errors with the full Vault response: permission denied (TokenReview refused or invalid token), invalid role name, namespace not authorized, or a 403 on the secret path when the policy does not cover the data/ path the annotation names.

Injector or External Secrets Operator?

The injector keeps secrets out of the Kubernetes API: files in an in-memory volume, renewed by a sidecar, good for dynamic secrets and apps that read files. ESO writes a normal Secret, which charts and other tools expect and which works without a sidecar per pod, but the value then sits in etcd and anyone allowed to read Secrets in the namespace can read it. Many platforms run both.

Why does the secret file contain "data: map[...]"?

Because without an agent-inject-template-<name> annotation the injector renders the whole secret with Go's default formatting. For KV v2 that is the data and metadata maps. Add a template that prints exactly the lines your app reads, using .Data.data.KEY for KV v2 values (one level deeper than for dynamic secrets).

In an interview Mid

How does a pod in Kubernetes get a secret from Vault without a token baked into its manifest?

It proves its identity with the ServiceAccount token Kubernetes already mounts in every pod. Vault's kubernetes auth method receives that JWT on auth/kubernetes/login with a role name, sends a TokenReview to the API server to confirm it is valid and whose it is, then checks the role: is the ServiceAccount name in bound_service_account_names and the namespace in bound_service_account_namespaces? If yes it issues a short-lived Vault token with the role's policies.

The reviewer must be allowed to create TokenReviews (system:auth-delegator). Delivery is then the Agent Injector (annotations on the pod template; an init container and sidecar render files under /vault/secrets), ESO (a Kubernetes Secret), the CSI driver, or the app calling Vault itself.

Also asked: What is "secret zero" and how do platforms avoid storing it? · What are the trade-offs of syncing Vault secrets into Kubernetes Secrets? · A pod fails to start after you added Vault annotations. How do you debug it?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.