OnCallReadyStart learning

Security and secrets Lesson 24.24 · Vault & Secrets Management · 24 min read

Vault Agent: auto-auth, templates, and apps that never see Vault

In plain words

Imagine a personal assistant who holds your building badge for you. Each morning they swipe in, fetch the documents you need, put them on your desk, and when a document changes they swap in the new copy. When the badge is about to expire they get a new one, without you noticing.

Vault Agent is that assistant for an app. With auto-auth it logs in (AppRole, Kubernetes), keeps the token renewed and logs in again before the max TTL. Its templates fetch secrets and render them into files in exactly the format the app reads, re-rendering when a secret or a dynamic credential changes, and can run a command so the app reloads. The app itself never talks to Vault at all.

Why this lesson exists

Everything so far needed the client to do work: log in, keep the token renewed, log in again before the max TTL, fetch new database credentials when the lease ends, reconnect. Most applications do none of that. They read a password at start-up and keep it forever - and fail, at 3 a.m., the day the token or the lease runs out. Vault Agent is the answer HashiCorp ships: a small daemon next to the application that handles Vault, so the application only reads a file. This lesson is the agent's configuration, its log, and the failure modes you will be paged for.

What you need to know already: AppRole logins and secret IDs (the AppRole lesson); TTLs, renewal and re-authentication (the tokens lesson); leases (the dynamic secrets lesson); systemd units and journalctl -u (2.5, 2.30). The Go template syntax the agent uses ({{ }}) is explained here.

The words you need first

  • Vault Agent - vault agent -config=FILE: the same vault binary, running as a client-side daemon.
  • Auto-auth - the agent logs in by itself with a configured auth method (AppRole on a VM, Kubernetes in a pod, a cloud identity), renews the token, and logs in again when the token cannot be renewed any more.
  • Sink - where the agent writes the token it got (a file), for programs that want a token rather than secrets.
  • Template - a file the agent renders from Vault data: config files, .env files, certificates. The syntax is consul-template's, a Go template.
  • Render - the agent fetched the secrets and wrote the destination file.
  • Lifetime watcher - the part of the agent that renews a token or lease, and signals when renewal can no longer extend it.

What the agent does for you

              without an agent                      with Vault Agent
app logs in itself (needs secret zero)       agent logs in (AppRole / Kubernetes / ...)
app renews its token... or forgets           agent renews; re-authenticates at the max TTL
app fetches DB creds once at start           agent re-fetches before the lease ends
app holds a Vault client library             app reads /etc/app/db.env - no Vault code
token expires -> permission denied           a new file appears; app re-reads (or is told)

The application's only contract is a file. Vault knowledge lives in one config file the platform team reviews.

The configuration

# /etc/vault-agent/agent.hcl
vault {
  address = "https://127.0.0.1:8200"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path                   = "/etc/vault-agent/role-id"
      secret_id_file_path                 = "/etc/vault-agent/secret-id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "/run/vault-agent/token"
      mode = 0640
    }
  }
}

template {
  source      = "/etc/vault-agent/db.env.tpl"
  destination = "/etc/orders-sync/db.env"
  perms       = "0640"
}
  • vault.address - where Vault is; ca_cert if its certificate is not in the system trust store.
  • auto_auth.method - here AppRole, reading the role ID and secret ID from files. Note remove_secret_id_file_after_reading: its default is true - the agent deletes the secret ID file after the first login (the secret ID is single-use anyway). A restarted agent then cannot log in again unless something delivers a new secret ID. For a long-lived VM service with a reusable secret ID, set it to false - and protect the file.
  • sink - optional; the token is written there for scripts that need one.
  • template - one per file. source is a template file (or contents inline), destination the rendered file, perms its mode.

The template:

{{ with secret "database/creds/orders-ro" -}}
DB_USER={{ .Data.username }}
DB_PASSWORD={{ .Data.password }}
{{- end }}
  • {{ with secret "PATH" }} - read PATH from Vault (with the agent's token) and use the response inside the block.
  • .Data is the response's data. For KV v2 the secret is one level deeper: {{ with secret "secret/data/orders-sync/config" }}{{ .Data.data.api_key }}{{ end }} - the double .Data.data again.
  • {{- and -}} trim the whitespace around an action, so the file has no blank lines.
  • A missing key renders as <no value> - silently - unless error_on_missing_key = true in the template stanza. Turn it on: a broken config should fail loudly.

Running it

As a service, next to the app:

# /etc/systemd/system/vault-agent.service (an illustration)
[Unit]
Description=Vault Agent for orders-sync
After=network-online.target vault.service
Wants=network-online.target

[Service]
ExecStart=/usr/bin/vault agent -config=/etc/vault-agent/agent.hcl
Restart=on-failure

[Install]
WantedBy=multi-user.target

Or by hand in the background to watch it work:

$ sudo vault agent -config=/etc/vault-agent/agent.hcl &
[1] 17386
==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v2.2.0, built 2026-10-14T19:24:08Z
             Version Sha: 2891bfd23893ebd302307050edaf03d871ea8351

2026-09-22T20:00:04.000Z [INFO]  agent.sink.file: creating file sink
2026-09-22T20:00:04.000Z [INFO]  agent.sink.file: file sink configured: path=/run/vault-agent/token mode=-rw-r----- owner=0 group=0
2026-09-22T20:00:04.000Z [INFO]  agent.auth.handler: starting auth handler
2026-09-22T20:00:04.000Z [INFO]  agent.template.server: starting template server
2026-09-22T20:00:04.000Z [INFO]  agent.sink.server: starting sink server
2026-09-22T20:00:04.000Z [INFO]  agent.auth.handler: authenticating
2026-09-22T20:00:04.000Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2026-09-22T20:00:04.000Z [INFO]  agent.sink.file: token written: path=/run/vault-agent/token
2026-09-22T20:00:04.000Z [INFO]  agent.auth.handler: starting renewal process
2026-09-22T20:00:04.000Z [INFO]  agent.template.server: template server received new token
2026-09-22T20:00:04.000Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2026-09-22T20:00:04.000Z [INFO]  agent: (runner) creating watcher
2026-09-22T20:00:04.000Z [INFO]  agent: (runner) starting
2026-09-22T20:00:04.000Z [INFO]  agent: (runner) rendered "/etc/vault-agent/db.env.tpl" => "/etc/orders-sync/db.env"

Read the log top-down: auth handler (logged in), sink (token written), renewal process started, the template runner started and rendered the file. Every Vault Agent problem shows up as one of these steps not happening.

What happens over time

Token renewal and re-authentication

The auth handler renews the token at about two thirds of its TTL. When a renewal comes back capped - the max TTL is near - the lifetime watcher ends and the agent logs in again:

2026-09-22T20:00:43.700Z [INFO]  agent.auth.handler: renewed auth token
2026-09-22T20:01:23.700Z [INFO]  agent.auth.handler: renewed auth token
2026-09-22T20:02:43.700Z [INFO]  agent.auth.handler: renewed auth token
2026-09-22T20:02:53.700Z [INFO]  agent.auth.handler: lifetime watcher done channel triggered, re-authenticating
2026-09-22T20:02:53.700Z [INFO]  agent.auth.handler: authenticating
2026-09-22T20:02:53.700Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks

That last block is the step an application with a hard-coded token never takes. It needs the auth method's credentials to still work: with AppRole, a secret ID that is still valid (or still on disk - see remove_secret_id_file_after_reading).

Static and dynamic secrets in templates

  • Static secrets (KV) are re-read every static_secret_render_interval, 5 minutes by default (in a template_config block). A KV change reaches the file within 5 minutes, not instantly.
  • Dynamic secrets (database credentials, PKI certificates) are renewed while their lease allows; when the lease cannot be renewed further, the agent fetches new credentials before the old ones expire and renders the file again:
2026-09-22T20:00:03.600Z [INFO]  agent: (runner) rendered "/etc/vault-agent/db.env.tpl" => "/tmp/db.env"
2026-09-22T20:05:33.600Z [INFO]  agent: (runner) rendered "/etc/vault-agent/db.env.tpl" => "/tmp/db.env"

(here with a role whose max_ttl is 6 minutes: new credentials 30 seconds before the old lease ends). A new file is worthless if the application never reads it again. Three ways to close that gap:

  1. the application re-reads the file on each connection or on a timer (simplest);
  2. the template stanza's command (or exec { command = [...] }) runs something after each render: systemctl reload orders-sync;
  3. the agent's process supervisor mode (an exec block plus env_template): the agent starts the application itself with the secrets in its environment, and restarts it when they change.

Errors in the log

The agent keeps running and retries, with backoff, when something fails - which means a broken agent looks alive. Learn the three lines:

[ERROR] agent.auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://127.0.0.1:8200/v1/auth/approle/login
  | Code: 400. Errors:
  |
  | * invalid role or secret ID
   backoff=2s

Login fails: wrong or used-up secret ID, wrong role ID, CIDR binding, Vault sealed (a 503 Vault is sealed in the same place).

[WARN]  agent: (view) vault.read(database/creds/orders-ro): vault.read(database/creds/orders-ro): Error making API request. URL: GET https://127.0.0.1:8200/v1/database/creds/orders-ro Code: 403. Errors: * 1 error occurred: * permission denied (retry attempt 3 after "1000ms")

Logged in, but the token's policy does not allow the template's path - the KV v2 data/ mistake shows up exactly here.

[ERROR] agent: (runner) watcher reported error: failed writing file: open /etc/orders-sync/db.env: permission denied

Rendered, but the agent's user may not write the destination.

Agent, proxy, or library?

  • Vault Agent - templates and auto-auth for applications that read files or environment variables. The default choice for VMs and the basis of the Kubernetes Agent Injector (next lesson).
  • Vault Proxy (vault proxy) - auto-auth plus an API proxy with caching, for applications that already speak the Vault API: they send requests without a token and the proxy adds its own.
  • A Vault client library in the application (Spring Cloud Vault, hvac for Python) - most control, most code; the application owns renewal and re-authentication.

How to choose, in practice:

the application...use
reads a config file or environment variables, and you cannot change itthe agent with a template (and command to reload it)
must get secrets as environment variables at startthe agent's process supervisor mode: an exec stanza runs the app as a child and env_template blocks set its environment; the agent restarts it when a secret changes
already calls the Vault API itselfVault Proxy, so the app stops holding its own token
needs encryption (transit) or signing in its own codea client library, with the proxy or agent handling the token

Whatever you pick, the rule from the tokens lesson stays: somebody must log in again before the max TTL, and somebody must fetch new dynamic credentials before their lease runs out. The agent and the proxy do both for you; a library only does them if the application's code does. A common review question is simply "who renews, and who logs in again?" - if the answer is "the app, we think", that is the next 3 am incident.

One more option exists on Kubernetes only: the Vault CSI provider, which mounts secrets as a volume through the Secrets Store CSI driver without a sidecar per pod. It reads at pod start and on a rotation interval, but it does not render templates the way the agent does. The next lesson compares it with the injector and External Secrets.

In an interview: "An application reads its database password from Vault at start-up and stops working after a few weeks. What happened and how do you fix it properly?" - its token (or the credentials' lease) reached its max TTL; renewal can extend a token only up to the max, after that it must log in again, which the app never does. Fix it structurally with Vault Agent: auto-auth with AppRole or Kubernetes re-authenticates before the max, templates render the credentials to a file and re-render when they change, and the app re-reads the file (or the agent's command reloads it).

What you can now do

  • Write an agent config: vault address, AppRole auto-auth, a file sink, a template.
  • Write consul-template templates for KV v2 (.Data.data) and dynamic secrets (.Data).
  • Read the agent's log: auth, sink, renewal, render - and the three error lines.
  • Explain renewal versus re-authentication, static vs dynamic re-renders, and how the app learns about a new file.

Why it helps

Most Vault outages in applications come from the client side: tokens that hit their max TTL, leases that are never renewed, credentials read once at start-up and never again. Vault Agent solves those once, outside the app, which is why it is the recommended way to consume Vault and the engine behind the Kubernetes injector in the next lesson.

Reading the agent's log is also an on-call skill: "renewed auth token", "lifetime watcher done", "re-authenticating", a template error with the path and 403. The same lines appear in an injected pod's vault-agent container.

On the blog

Commands in this lesson

vault cat

FAQ

Why not let the app talk to Vault directly?

It can, with a Vault SDK, and some apps do. But then every app must implement login, renewal, re-authentication and lease handling correctly, in every language. The agent does it once, and the app only reads a file, which also works for software you cannot change.

What is a sink?

A place where auto-auth writes the token it holds, usually a file such as /run/vault-agent/token with tight permissions. Tools that need a token read it from there. If only templates use the token, you do not need a sink at all, which keeps the token out of the filesystem.

What happens when a dynamic secret in a template expires?

The agent renews the lease while it can. When it can no longer be renewed (max TTL), the agent fetches new credentials, re-renders the file and runs the template's command, so the app picks up the new username and password. Static KV values are re-read on a timer (five minutes by default).

Why does my template file say <no value>?

The template asked for a field that is not there. For KV v2 the values are one level deeper: .Data.data.password, not .Data.password. For dynamic secrets it is .Data.username. A wrong path, or a token without read on it, also produces an empty result or an error in the agent's log.

Agent or proxy mode?

Agent mode renders templates and keeps a token for the app. Vault Proxy, split out of the agent, listens locally and forwards the app's own API calls with the auto-auth token, optionally caching. Use the agent when the app reads files, the proxy when it already calls the Vault API.

In an interview Mid

An application reads its database password from Vault at start-up and stops working after a few weeks. What happened, and how do you fix it properly?

Its token, or the credential's lease, reached its max TTL. Renewal extends a token only up to the max; after that it must log in again, which the app never does, and when the token expires its leases are revoked, so the database user is dropped.

The structural fix is Vault Agent: auto-auth with AppRole or Kubernetes renews the token and re-authenticates before the max, and templates render the credentials to a file and re-render when they change - a renewed lease, new dynamic credentials - running a command so the app reloads. The app reads a file and never handles tokens.

Also asked: How would you run Vault Agent next to a service on a plain VM? · What does the agent do when Vault is unreachable for a while? · How do you make an application reload when its secrets file changes?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.