Secret Management for Production Kubernetes Workloads

etcd stores Kubernetes Secrets as base64 text, and that gap is the single most common finding I see when I go through a startup's infrastructure. Anyone with kubectl access can decode a "protected" secret in one command. No key is required, and no log entry is generated. Closing that gap takes a few layers stacked on top of each other, and most teams only build the first one.

Base64 is a text-encoding scheme, nothing more. It turns binary data into a string safe to drop into YAML or pass through an API call. It was never built to hide anything. Run echo <string> | base64 -d and the original value comes right back, no password, no cryptographic step. Real encryption needs a key to reverse it. Base64 needs nothing.

So in practice, a Secret object sits in etcd as plain base64 text. It shows up in API server logs. It shows up when someone runs kubectl describe against the wrong resource by accident. It shows up in a CI/CD log dump when a pipeline prints its full environment for debugging, which happens more than you'd think. None of that requires a breach. It just requires someone with normal, legitimate access doing something completely ordinary.

The scale backs this up too. GitGuardian counted 28.65 million new hardcoded secrets in public GitHub commits during 2025, up 34% from the year before, the sharpest jump they've recorded. Most of those weren't planted by attackers. A developer pastes a real API key into a test file, commits it, moves on, and doesn't think about it again until something goes wrong.

Part of this comes down to the name. Kubernetes calls the object "Secret," and that word does a lot of quiet damage. Teams treat the problem as solved because the platform handed them a reassuring label rather than actual protection.

What etcd encryption at rest actually does and how to turn it on

Encryption at rest means kube-apiserver encrypts a Secret object before writing it to etcd, then decrypts it in memory when something asks for it back. That's the entire mechanism. It's not on by default, and a surprising number of production clusters never turn it on at all.

You enable it with an EncryptionConfiguration manifest passed to kube-apiserver at startup, and inside that file you pick a provider. The choice matters more than the YAML makes it look:

• identity: no encryption, full stop. This is the default.
• aescbc / aesgcm: symmetric encryption, key stored in the config file itself. Better than nothing, but the key sits on the same node as the data it's protecting, which caps how much protection you're actually buying.
• kms: key management handed off to AWS KMS, Google Cloud KMS, or Azure Key Vault. This is the one you want in production, because the key lives somewhere the cluster doesn't control.

Here's what teams miss constantly: flipping encryption on doesn't touch anything already sitting in etcd. Old Secret objects stay exactly as they were until something rewrites them. You have to force that rewrite, usually by re-applying every Secret in the cluster, before the new setting protects data that predates it.

Watch your backups too. An etcd snapshot taken before encryption was enabled, or taken while it was off, holds every secret in plain text, and it doesn't care what your current config says. I've seen this discovered mid-incident more than once, which is about the worst time to find out.

Encryption at rest also has a hard ceiling. It protects data sitting in etcd. It says nothing about who can read that data through the API, how it flows into a running pod, or what happens once it's an environment variable inside a container. Storage layer only.

RBAC and the access controls that determine who can read secrets through the API

Encrypt etcd all you want; any service account or user with get or list on the secrets resource still pulls back plain text through the API. Encryption locks the disk. RBAC decides who's allowed to ask for what's on it.

Most clusters get this wrong on day one and never come back to fix it. Someone sets up broad roles during initial setup because it's fastest, everything works, and nobody revisits permissions once the cluster's live and doing real work. That first pass becomes permanent by neglect, which is how most bad RBAC configs actually happen.

Least privilege for secrets comes down to a few concrete habits, and none of them are complicated:

• Workloads get access only to the specific secrets they need, scoped to their own namespace, not the cluster.
• Cluster-wide read access to secrets should belong to almost nothing. If a ClusterRole grants it, assume it's a mistake until someone proves otherwise.
• Keep the people or pipelines that create and update secrets separate from the pods that just read them. CI creates, apps read. Different jobs, different permissions.

Audit logging matters as much as the RBAC rules themselves. Turn on logging for get and list against secrets. Skip that step, and there's no record of who read what and when, so any forensic work after an incident hits a wall almost immediately.

Service account hygiene closes this out. Don't mount the default service account token into pods that never call the Kubernetes API; set automountServiceAccountToken: false and remove the risk entirely. Where a workload needs cloud permissions, use workload identity, IRSA on EKS or Workload Identity on GKE, so permissions route through cloud IAM instead of a long-lived secret sitting in the cluster.

RBAC governs the API. What an app does with a secret once it's holding one is a separate risk entirely, and RBAC has nothing to say about it.

Why environment variables are a weak injection mechanism and what to use instead

Env vars are the default almost everyone reaches for, mostly because they're dead simple. They're also leaky in ways that don't show up until something goes wrong.

Every process in the container can read them, straight from /proc/[pid]/environ. Logging frameworks, crash reporters, and APM agents routinely grab the full environment automatically, sometimes shipping it to a third-party monitoring vendor without anyone asking them to. Run printenv during a debug session and every secret in that container lands on stdout, where a log aggregator picks it up and stores it, sometimes for months.

Volume mounts are the real fix. Mount the secret as a file into the pod's filesystem instead. The app reads the file at startup or on demand, and the value never touches the process environment. Kubernetes will even refresh the mounted file automatically when the Secret changes upstream, no pod restart needed, though there's a short delay before the new value shows up.

Here's the part that actually matters more: whether you inject the secret as an env var or a mounted file, it still lives in etcd as a Kubernetes Secret object either way. The cluster stays the single source of truth. As the number of services grows, that turns into a sprawl problem on its own: secrets scattered across dozens of namespaces with nobody holding a central view of what exists or who's touching it.

That sprawl is what pushes teams toward pulling the source of truth outside the cluster completely.

The external secret store pattern that has become the 2025–2026 production standard

Store secrets in a dedicated platform outside Kubernetes, sync them into the cluster only when something actually needs them. The cluster stops being the vault and becomes a consumer.

Microservice architecture is what forced this shift. Credentials end up scattered across git history, .env files, Slack DMs, CI logs, wherever a developer happened to be sitting when they needed a value fast. Better local hygiene only gets you so far; pulling local storage out of the picture entirely fixes the actual root cause instead of managing around it.

The credential model changed alongside the storage model. Long-lived secrets that get rotated every few months tend to pile up across a dozen systems nobody's tracking closely, so teams have moved to just-in-time access: short-lived credentials, issued on demand, used once or for a short window, then gone. Access shifts to identity, an app proves who it is through an AWS IAM role or a Kubernetes Service Account, and the only question that matters is who's asking and whether they're allowed.

By 2025 and into 2026 the common stack looks pretty settled: a KMS holds the root encryption key, a cloud secret manager or Vault holds the actual secrets, and the External Secrets Operator delivers them into the cluster.

One thing worth saying plainly: this doesn't remove etcd from the picture. Secrets still land in etcd as Kubernetes Secret objects once the External Secrets Operator syncs them in. Encryption and RBAC don't become optional because you bolted on an external store. This pattern is a layer on top of the foundation, and the foundation still has to hold its own weight.

How the main external secret tools differ and when to reach for each

External Secrets Operator (ESO) reads from an external store and writes standard Kubernetes Secret objects into the cluster. Teams already on AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault reach for it when they want those secrets delivered into Kubernetes without standing up anything new. It's got a real community behind it, over 6,000 GitHub stars and more than 1,100 forks as of mid-2026. Same caveat as always applies though: ESO writes into etcd, so encryption and tight RBAC don't go away. It fits naturally into GitOps and keeps app code decoupled from any one platform's secrets API.

HashiCorp Vault is a full platform. It issues short-lived credentials, rotates them, enforces access policies, logs every access event. Its Transit engine lets apps encrypt and decrypt data through API calls without ever touching the raw key, which I've found genuinely useful for protecting PII or payment data at the application layer, not just the infrastructure layer. The Vault Secrets Operator injects secrets directly into pods without persisting them, which sidesteps the etcd residue problem entirely. Vault fits fintech, healthcare, crypto, anywhere a full audit trail and platform independence aren't up for debate. The cost is real, though: Vault needs its own infrastructure, its own HA setup, and ongoing care that's a meaningful lift for a small team to carry.

AWS Secrets Manager is the managed route: store, retrieve, get automatic rotation for RDS, Redshift, and DocumentDB, all wired tightly into AWS already. It's a good fit for AWS-native workloads that don't need dynamic short-lived credentials, and it pairs with ESO or the AWS Secrets and Configuration Provider to get into Kubernetes. Less operational burden than running Vault yourself, but you're trading that for AWS lock-in and less room to move if you ever go multi-cloud.

Doppler and Infisical are the newer wave, built developer-experience-first and now genuinely enterprise-capable, with native CI/CD integrations and OIDC-based pipeline auth. Infisical is fully open-source under MIT. OpenBao, a community fork of Vault under MPL 2.0, is worth watching if you want Vault's capability set without Vault's licensing terms. Doppler and Infisical suit smaller teams that want managed simplicity and don't want to carry the operational weight of self-hosted Vault.

Rough guide, for what it's worth: AWS-native without dynamic secrets, pair Secrets Manager with ESO. Multi-cloud or compliance-heavy, look at Vault or OpenBao. Small team, developer experience first, Doppler or Infisical. GitOps-first with secrets encrypted right in Git, that's SOPS or Sealed Secrets, which I'll get into next.

Managing secrets in GitOps pipelines without committing plaintext to Git

GitOps wants Git holding the single source of truth for everything, config included. Secrets don't fit that model cleanly, and committing them in plain text is still one of the most common failures I run into in DevOps audits.

SOPS (Secrets OPerationS) is a CNCF sandbox project now, and it recently added age as a first-class encryption backend alongside PGP and cloud KMS. It encrypts secret files before they ever touch Git. Flux has native SOPS support; ArgoCD needs a plugin to get there, since it isn't built in. SOPS has become the default for file-level secret encryption in GitOps. It works well for small teams keeping encrypted files right in the repo, but it leans entirely on strong external KMS hygiene: get that wrong and the key becomes the exact single point of failure you were trying to design around.

Sealed Secrets takes a different route: encrypt a Secret manifest locally with a cluster-specific public key, commit the SealedSecret to Git, and let an in-cluster controller decrypt it back into a live Secret. Self-contained, nothing external needed at runtime, which appeals to air-gapped setups or clusters that want everything under GitOps with nothing reaching outside the cluster. The pitfall is sharp, though: the controller's private key belongs to that specific cluster. Lose the controller without a backup of that key, and every sealed secret in your repo becomes permanently unreadable, forever. Some teams respond by keeping plaintext copies "just in case," which quietly defeats the whole point of using Sealed Secrets to begin with.

ArgoCD and Flux both need credentials to reach the Git repo at bootstrap, and those credentials often carry full read access. They end up stored as cluster Secrets, which makes them high-value targets in their own right, easy to overlook because they feel like plumbing rather than data. Use OIDC or short-lived tokens wherever the Git provider allows it, and audit these bootstrap credentials with the same rigor you'd give an application secret.

On the CI/CD side, OIDC-based pipeline auth, GitHub Actions paired with AWS OIDC is the common example, removes long-lived static credentials from CI entirely. Pre-commit scanning with something like GitGuardian catches hardcoded credentials before they land in the codebase at all, and that matters because the most common failure mode I still see is a developer pasting a real credential into a file during local testing and just forgetting it's sitting there.

How secret management maps to SOC 2 and HIPAA requirements in Kubernetes environments

Secret management is the technical substance behind several control categories SOC 2 and HIPAA both care about, and any auditor worth their fee can tell the difference between a real control and a policy document nobody follows.

Under SOC 2, CC6 covers logical and physical access controls. RBAC on secrets, audit logging of secret access, least-privilege service accounts, that's direct evidence here. An auditor asking for proof access is restricted wants the RBAC policy and logs showing it's actually enforced, not just a written policy describing good intentions.

CC7 covers system operations, and that's where etcd encryption, KMS-backed key management, and documented rotation policies map onto monitoring and change management. Auditors ask for evidence of automated rotation and continuous access logging now, more than they used to, and a manual rotation process run whenever someone happens to remember doesn't hold up against that anymore. Vault's dynamic credentials, ESO's sync-based delivery, KMS-backed etcd encryption: these carry real operational weight. They shape the difference between an audit that takes a week and one that drags on for a month because nobody can answer a simple question about who touched a given secret and when.