OnCallReadyStart learning

Security and secrets Lesson 24.21 · Vault & Secrets Management · 28 min read

PKI and transit: certificates on demand, encryption as a service

In plain words

Think of a passport office. The government (the root CA) does not stamp every passport itself; it authorises regional offices (intermediate CAs), and those issue passports with an expiry date. Each office has rules about who may get what (roles). If passports only last a few days, a stolen one is soon worthless and nobody needs a long list of cancelled passports.

Vault's PKI engine is a passport office for TLS certificates: a root, an intermediate it signs, roles that limit names and lifetimes, and issue that returns a certificate and key in one call. The transit engine is different: a notary who stamps and unstamps documents for you, so you store only the stamped (encrypted) version and never hold the key yourself.

Why this lesson exists

Two more things Vault does that have nothing to do with storing passwords. PKI: it is a certificate authority, so every internal service can get a TLS certificate in one API call - valid for days, not years, so expiry stops being a yearly outage. Transit: it encrypts and decrypts data for applications that never see the key, so a stolen database dump of card numbers is ciphertext. Both come up in bank platform teams constantly, and both have operational gotchas of their own.

What you need to know already: certificates, chains, CAs and openssl x509 (11.15, 11.17); the expiry incident from the TLS chapter; leases and -field (earlier in this chapter); base64 (17.33).

The words you need first

  • CA (certificate authority) - signs certificates. A root CA signs itself; an intermediate CA is signed by the root and signs the leaf certificates.
  • Issuer - Vault's name for a CA certificate + key inside a PKI mount. A mount can hold several (for rotation); one is the default.
  • PKI role - the rules for what a mount may issue: which domain names, how long, which key type.
  • CSR (certificate signing request) - a request containing a public key and a name, sent to a CA to be signed.
  • CRL (certificate revocation list) - the CA's signed list of certificates it has revoked before their expiry.
  • Transit - Vault's "encryption as a service" engine: you send plaintext, you get ciphertext, the key never leaves Vault.
  • Key version - transit keys rotate: each rotation adds a version; data encrypted with old versions still decrypts until you forbid it.

PKI: a two-level CA in Vault

The shape every guide recommends: a root CA that only signs intermediates (and in a real company lives offline or in its own mount, rarely touched), and an intermediate that issues the day-to-day certificates. If the intermediate is ever compromised you revoke and replace it; the root - which every client trusts - stays.

The root

# enabling the two mounts (the lab's setup has done it - an illustration, no ▶)
$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/
$ vault secrets enable -path=pki_int pki
Success! Enabled the pki secrets engine at: pki_int/
$ vault secrets tune -max-lease-ttl=87600h pki
Success! Tuned the secrets engine at: pki/
$ vault write -field=certificate pki/root/generate/internal common_name="oncall-lab Root CA" issuer_name=root-2026 ttl=87600h > /tmp/root_ca.crt
$ openssl x509 -in /tmp/root_ca.crt -noout -subject -issuer -dates
subject=CN=oncall-lab Root CA
issuer=CN=oncall-lab Root CA
notBefore=Sep 22 19:59:33 2026 GMT
notAfter=Sep 19 20:00:03 2036 GMT
  • tune -max-lease-ttl first: a PKI mount cannot issue anything longer than its max lease TTL, and the default is 768h (32 days) - too short for a CA.
  • generate/internal - the private key is created inside Vault and never leaves it (generate/exported would return it once). For a root you usually want internal.
  • Subject = issuer: self-signed, ten years.

The intermediate

$ vault secrets tune -max-lease-ttl=43800h pki_int
$ vault write -field=csr pki_int/intermediate/generate/internal common_name="oncall-lab Intermediate CA 2026" > /tmp/int.csr
$ vault write -field=certificate pki/root/sign-intermediate csr=@/tmp/int.csr format=pem_bundle ttl=43800h > /tmp/int.crt
$ vault write pki_int/intermediate/set-signed certificate=@/tmp/int.crt
Key                 Value
---                 -----
existing_issuers    <nil>
existing_keys       <nil>
imported_issuers    [59281aeb-c3f8-e17a-0d4c-e2748adce9ad]
imported_keys       [634fdb3e-513a-073a-ffa6-05fca105930d]
mapping             map[59281aeb-c3f8-e17a-0d4c-e2748adce9ad:634fdb3e-513a-073a-ffa6-05fca105930d]

The standard four steps: the intermediate mount makes a key and a CSR, the root signs it (five years), the signed certificate goes back to the intermediate mount, which now has an issuer paired with its key. Exactly what you would do with openssl by hand (11.17) - except no private key ever touched a disk.

Roles: what may be issued

$ vault write pki_int/roles/internal allowed_domains=oncall-lab.internal allow_subdomains=true max_ttl=72h
Success! Data written to: pki_int/roles/internal

A role is the policy of the CA: *.oncall-lab.internal names only, at most 72 hours. Who may use the role is a Vault policy on pki_int/issue/internal - the orders team gets update on that path and nothing else.

Issuing

$ vault write -format=json pki_int/issue/internal common_name=orders.oncall-lab.internal alt_names=orders-api.oncall-lab.internal ttl=24h > /tmp/orders.json
$ jq -r .data.certificate /tmp/orders.json > /tmp/orders.crt
$ jq -r .data.private_key /tmp/orders.json > /tmp/orders.key
$ openssl x509 -in /tmp/orders.crt -noout -subject -issuer -dates -ext subjectAltName
subject=CN=orders.oncall-lab.internal
issuer=CN=oncall-lab Intermediate CA 2026
notBefore=Sep 22 19:59:34 2026 GMT
notAfter=Sep 23 20:00:04 2026 GMT
X509v3 Subject Alternative Name:
    DNS:orders.oncall-lab.internal, DNS:orders-api.oncall-lab.internal

The response holds everything a server needs: certificate, private_key, issuing_ca, ca_chain, serial_number, expiration. The private key is generated for this request and not stored by Vault - save it now or issue again. (sign/<role> signs a CSR instead, when the key must be generated on the server itself.)

Note the notBefore thirty seconds in the past: Vault backdates certificates slightly so a client whose clock is a little behind does not reject a brand-new certificate as "not yet valid".

The role enforces its rules:

$ vault write pki_int/issue/internal common_name=orders.example.com
Error writing data to pki_int/issue/internal: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/pki_int/issue/internal
Code: 400. Errors:

* 1 error occurred:
	* common name orders.example.com not allowed by this role

A TTL above the role's max is cut down, with a warning:

* TTL "720h" is longer than permitted maxTTL "72h", so maxTTL is being used

And nothing outlives its CA: asking for a certificate that would expire after the issuing CA does fails with cannot satisfy request, as TTL would result in notAfter ... that is beyond the expiration of the CA certificate.

Short certificates, revocation, CRLs

The point of a Vault PKI is short-lived certificates: 24-72 hours, renewed automatically (Vault Agent templates, cert-manager's Vault issuer in Kubernetes). A certificate that expires tomorrow does not need revoking when a server is decommissioned

  • and an expiry outage becomes impossible, because renewal runs all the time, not once a

year.

When you do need to revoke - here the orders certificate from above, by its serial:

$ vault write pki_int/revoke serial_number=$(jq -r .data.serial_number /tmp/orders.json)
Key                        Value
---                        -----
revocation_time            1790107224
revocation_time_rfc3339    2026-09-22T20:00:24.802207038Z
state                      revoked

The serial goes on the CRL, which clients fetch - without a token - from /v1/pki_int/crl/pem (and the CA itself from /v1/pki_int/ca/pem). Revocation only works for clients that check the CRL (or OCSP); many do not. Another reason to prefer short lifetimes. vault list pki_int/certs lists serials of issued certificates (unless the role has no_store); vault write pki_int/tidy tidy_cert_store=true tidy_revoked_certs=true cleans out expired ones, which matters on a busy mount.

Transit: encryption as a service

An application must store card numbers (PANs) in its database. Encrypting them in the app means the app holds the key - and so does every backup of its config. With transit the app sends the plaintext to Vault and stores what comes back.

# enabling it (the lab's setup has done it - an illustration, no ▶)
$ vault secrets enable transit
Success! Enabled the transit secrets engine at: transit/
$ vault write -f transit/keys/cards
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1790107204]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      cards
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96

aes256-gcm96 is the default key type. exportable false - the key can never be read out of Vault; deletion_allowed false - it cannot be deleted by accident (that would make every ciphertext garbage).

Transit works on base64, so binary data travels safely in JSON:

$ vault write transit/encrypt/cards plaintext=$(echo -n "4111 1111 1111 1111" | base64)
Key            Value
---            -----
ciphertext     vault:v1:27nN3cSz27F++EXmwFzJhtNcxhXHTcrsxFzae8Fcw2vvbfDY8m31kfFt8/5PRc0=
key_version    1

vault:v1: says which key version encrypted it. Decrypting gives base64 back:

$ CT=$(vault write -field=ciphertext transit/encrypt/cards plaintext=$(echo -n "4111 1111 1111 1111" | base64))
$ vault write -field=plaintext transit/decrypt/cards ciphertext=$CT | base64 -d; echo
4111 1111 1111 1111

Forget the base64 step and Vault refuses: * failed to base64-decode plaintext. Forget it on the way back and you store NDExMSAxMTEx... and wonder why the numbers look odd.

That also means you cannot search the database for a card number by its ciphertext. If you need lookups, store an HMAC alongside (transit/hmac/cards), which is deterministic.

Rotating and rewrapping

$ vault write -f transit/keys/cards/rotate
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1790107203 2:1790107232]
latest_version            2
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      cards
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96
$ vault write transit/rewrap/cards ciphertext=$CT
Key            Value
---            -----
ciphertext     vault:v2:b2OtXs8zI5Q5AJVWq39Hf4B/RtCsbkIZr39UipJ/TZqkTn4toU55dKJOewc1ANE=
key_version    2
  • rotate adds version 2; new encryptions use it, old vault:v1: ciphertext still decrypts.
  • rewrap re-encrypts a ciphertext with the newest version without ever returning the plaintext - a batch job can walk the table and rewrap every row, and the job never sees a card number.
  • Once every row is v2, retire version 1: vault write transit/keys/cards/config min_decryption_version=2. From then on v1 ciphertexts fail with * ciphertext or signature version is disallowed by policy (too old) - check the table first.

Policies for transit

The app needs exactly two paths:

path "transit/encrypt/cards" {
  capabilities = ["update"]
}
path "transit/decrypt/cards" {
  capabilities = ["update"]
}

Not transit/keys/cards (key management), not rotate. The batch job that rewraps gets transit/rewrap/cards - and no decrypt. Separation of duties in four lines.

In an interview: "How would you protect card numbers in a database so a stolen dump is useless?" - encrypt them through Vault's transit engine: the app sends base64 plaintext to transit/encrypt/<key> and stores the vault:v1:... ciphertext; the key never leaves Vault, the app's policy allows only encrypt/decrypt on that key, rotation adds versions and rewrap upgrades old rows without exposing plaintext.

What you can now do

  • Build a root and an intermediate CA in two PKI mounts, with the right max TTLs.
  • Write a PKI role and issue a certificate; read it with openssl; verify the chain.
  • Revoke a certificate and find the CRL; explain why short lifetimes beat revocation.
  • Encrypt and decrypt with transit (and the base64 step both ways).
  • Rotate a transit key, rewrap ciphertext, and retire old versions safely.
  • Write least-privilege policies for an app and a rewrap job.

Why it helps

Running an internal CA by hand means long-lived certificates, spreadsheets of expiry dates and outages when one is forgotten. Vault's PKI makes short-lived certificates practical, which is why it sits behind many service meshes and cert-manager setups. Reading a certificate's issuer, chain and dates with openssl is a daily on-call skill anyway.

Transit answers a different security question: how do you protect sensitive columns, like card numbers, so a stolen database dump is useless? Encrypting through Vault keeps the key out of the app and the database, and key rotation and rewrap are standard compliance requirements.

Commands in this lesson

vault openssl jq

FAQ

Why have an intermediate CA instead of issuing from the root?

So the root's key is used rarely and can be kept offline or tightly guarded. If an intermediate is compromised, you revoke it and create a new one signed by the same root, and clients that trust the root keep working. Rebuilding trust in a new root means updating every client.

Why are short-lived certificates better than revocation?

Revocation depends on clients checking CRLs or OCSP, which many do badly or not at all. A certificate valid for 72 hours expires before most revocation would even propagate. Clients renew automatically (Vault Agent, cert-manager), so short lifetimes cost nothing once renewal is in place.

What does a PKI role control?

What may be issued under it: allowed_domains and whether subdomains, bare domains or IP SANs are allowed, the key type and size, max_ttl, and key usages. A client with update on pki_int/issue/web can only get certificates the web role allows.

What does vault:v2: in a transit ciphertext mean?

The ciphertext was produced by version 2 of the transit key. Rotating the key adds a version; new encryptions use the latest one, while older ciphertexts still decrypt as long as their version is at or above min_decryption_version. transit/rewrap re-encrypts old ciphertexts with the newest version.

Why does transit want base64 plaintext?

Because the API is JSON and the plaintext can be any bytes, not only text. You encode before encrypting (base64 <<< "...") and decode after decrypting (base64 -d). Forgetting the decode is the most common transit mistake: the "plaintext" you get back is still base64.

In an interview Mid

How would you protect card numbers in a database so that a stolen dump is useless?

Encrypt them with Vault's transit engine: the app sends the base64 plaintext to transit/encrypt/<key> and stores only the returned vault:v1:... ciphertext; to read, it calls transit/decrypt/<key>. The key never leaves Vault, so the database and its backups hold nothing usable on their own.

The app's policy allows only update on encrypt and decrypt for that one key, and every call is audited. Rotation (transit/keys/<key>/rotate) adds versions; transit/rewrap upgrades old rows to the newest version without the app seeing the plaintext, and min_decryption_version retires old versions.

Also asked: Why would you use an intermediate CA in Vault rather than issuing from the root? · How do you check which CA issued a certificate and when it expires? · What is the difference between encryption in transit and encryption as a service?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.