Secrets management with Vault
Taking credentials out of repositories, CI variables and config files, and issuing them from one audited place with short lifetimes where possible.
Problem
Credentials were spread across CI variables, configuration files and individual laptops. Rotating a secret was risky because nobody was sure everywhere it was used.
Constraints
- Applications could not all be changed at once to read secrets differently.
- Pipelines and Kubernetes workloads needed access without long-lived static tokens.
Decisions
- Workload identity for authentication to Vault
- CI jobs and Kubernetes service accounts prove who they are, so there is no bootstrap secret to leak.
- Policies scoped per application and environment
- A compromised workload can only read its own secrets, and staging can never read production.
- Inject secrets at runtime rather than at build time
- Images and repositories stay free of credentials and can be shared or scanned safely.
Architecture
Vault
Highly available Vault cluster with audit logging enabled; policies managed as code.
- HashiCorp Vault
CI authentication
Pipeline jobs authenticate with signed job tokens and receive short-lived, scoped credentials.
- GitLab CI
- JWT auth
Workload authentication
Kubernetes service accounts authenticate to Vault; secrets are delivered to pods at runtime.
- Kubernetes auth
- External Secrets or Vault Agent
Audit
Every read is logged with the identity that made it.
- Vault audit device
Build notes
Implementation
Vault policies, auth roles and secret engines were managed with Terraform so access changes went through review like any other infrastructure change. Applications migrated one at a time, starting with pipelines, which had the most credentials and the least runtime risk.
Outcome
Secrets have a single source and a clear owner. Access is scoped and auditable, and rotation no longer requires a search through repositories and servers.
What I learned
The hardest secret to remove is the one used to fetch all the others. Workload identity solved that.