Kubernetes Secrets Are Not Actually Secret: How to Harden Credential Storage in K8s

The Naming Is Misleading—and That's a Real Security Problem

When developers first encounter Kubernetes Secrets, the name implies protection. It doesn't deliver it by default. A Kubernetes Secret is, out of the box, a base64-encoded string stored in etcd. Base64 is an encoding, not encryption—anyone with read access to the etcd datastore, or sufficient RBAC permissions, can retrieve the plaintext value in seconds.

This gap between the implied promise and the actual behavior has led to a category of credential exposures that are easy to miss in security reviews. This article walks through exactly where K8s secrets leak, why the defaults create risk, and the concrete steps you can take to close the gaps.

Where Kubernetes Secrets Actually Live (and Leak)

1. etcd Is the Crown Jewel—and Often Left Unencrypted

All Kubernetes objects, including Secrets, are persisted in etcd. By default, they are stored in plaintext (base64-decoded). If an attacker gains access to an etcd snapshot—through a misconfigured backup bucket, an exposed etcd port, or a compromised node—they have every secret in the cluster.

Encryption at rest for etcd is available but must be explicitly enabled via an EncryptionConfiguration resource. It is not on by default in most self-managed clusters, and many teams never check.

2. Overly Permissive RBAC

Even with encryption at rest enabled, a cluster role that grants get, list, or watch on the secrets resource in the wrong namespace hands credentials to whoever holds that token. A common pattern that creates problems:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]

The list verb is particularly dangerous—it allows bulk enumeration of every secret in a namespace, not just a single named one. Many CI/CD service accounts are provisioned with this because it's the easiest way to get things working quickly.

3. Secrets Mounted as Environment Variables

Kubernetes lets you inject a secret value directly into a pod as an environment variable. This is convenient but creates multiple exposure surfaces:

• Environment variables are visible in process listings (/proc/<pid>/environ) on the host node.
• Many application frameworks and crash reporters automatically dump environment variables to logs on startup or on error—sending your database password or API key to whatever logging sink you use.
• kubectl describe pod output does not redact environment variable names, making it easier to enumerate what secrets are in use.

Mounting secrets as files in a volume is generally safer, as the value is written to a tmpfs mount rather than the process environment.

4. Secrets Baked into ConfigMaps by Mistake

ConfigMaps are designed for non-sensitive configuration. But under time pressure, developers sometimes put connection strings, tokens, or credentials into a ConfigMap because "it's just config." ConfigMaps have no access controls beyond RBAC and are often treated as non-sensitive in audit reviews. The result is credentials in a resource that is frequently over-shared.

5. Helm Values Files Checked Into Git

Helm charts often accept secret values via values.yaml or --set flags. When a values.yaml containing a database password gets committed to a Git repo—even a private one—it becomes part of history. The secret now lives in the Kubernetes cluster and in version control, doubling the exposure surface.

Concrete Hardening Steps

Enable Encryption at Rest for etcd

Create an EncryptionConfiguration manifest and pass it to the API server with --encryption-provider-config. Use the aescbc or secretbox provider as a baseline, or—better—use a KMS provider backed by AWS KMS, GCP Cloud KMS, or Azure Key Vault so the encryption key itself never lives on the node.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

After enabling, run kubectl get secrets --all-namespaces -o json | kubectl replace -f - to re-encrypt existing secrets with the new provider.

Tighten RBAC to the Minimum Necessary

• Audit every ClusterRole and Role that includes secrets in its resource list.
• Replace broad list/watch on secrets with get scoped to named resources where possible.
• Use namespace-scoped Roles instead of ClusterRoles wherever a workload only needs access within one namespace.
• Regularly run kubectl auth can-i list secrets --as=system:serviceaccount:<namespace>:<sa-name> to verify service account permissions are what you expect.

Prefer Volume Mounts Over Environment Variables

Refactor pod specs to mount secrets as files rather than environment variables. For secrets that rotate, this also lets Kubernetes refresh the mounted file without restarting the pod (when optional: false and the secret is updated).

Adopt an External Secrets Operator

Tools like External Secrets Operator or the Secrets Store CSI Driver allow you to store the authoritative value in a proper secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) and sync or mount it into Kubernetes at runtime. The secret never needs to exist as a Kubernetes Secret object at all—eliminating the etcd exposure vector entirely.

Audit Helm Values and Chart Inputs

Enforce a policy that no values.yaml file committed to source control contains credentials. Use --set flags sourced from a secrets manager in your CI pipeline, or use a tool like Helm Secrets (backed by SOPS) to encrypt values files before committing. Scan your repositories for patterns that indicate secrets were committed—especially in Helm chart directories.

What Good Looks Like: A Hardened Baseline

1. etcd encrypted at rest using a KMS-backed provider.
2. RBAC: no service account has list on secrets cluster-wide; access is namespace-scoped and name-specific where possible.
3. Secrets mounted as files, not environment variables, in all production workloads.
4. Authoritative secrets stored in an external secrets manager; Kubernetes Secret objects are ephemeral synced copies.
5. Automated scanning on every pull request and nightly against running cluster configurations to catch drift.

Don't Assume Your Cluster Is Clean Today

The biggest risk isn't a new misconfiguration you're about to make—it's the one that's already in place and has never been audited. Clusters accumulate RBAC permissions, forgotten service accounts, and stale secrets over months of iteration. A scan that maps your current exposure to compliance frameworks like SOC 2 and HIPAA can surface the gaps that a manual review would miss.

To check whether your repositories or infrastructure-as-code files already contain exposed credentials heading into your Kubernetes environment, run a free GhostCred scan and get results in about 60 seconds.

Summary

Kubernetes Secrets provide a convenient API for passing sensitive values to workloads—but the security properties most teams assume come with them must be deliberately configured. Base64 encoding, permissive RBAC defaults, environment variable injection, and Helm values in Git are all real, active exposure paths. Closing them requires a layered approach: encrypt etcd, minimize RBAC, use volume mounts, delegate to external secrets managers, and scan continuously. None of these steps are especially complex individually—the challenge is doing all of them consistently across every cluster and namespace you operate.