Secrets Management in Production: Vault, External Secrets and 2026 Best Practices

A database password hardcoded into a .env file committed to Git. An API token pasted into a CI environment variable with no expiry. An SSH key shared over Slack between three developers. These scenarios are not textbook examples: they are the daily reality of most development teams in 2026, and every occurrence is a ticking time bomb.

Managing secrets in production is a solved problem from a technical standpoint. Mature tools exist, proven patterns are documented, and the standards are clear. Yet secret leaks remain among the most exploited attack vectors. This article walks through concrete solutions: from HashiCorp Vault to the Kubernetes External Secrets Operator, including automated detection and a complete operational checklist.

The problem: why secrets still leak

.env files and environment variables

The .env file has become the de facto standard for storing application configuration. The problem is twofold: it holds secrets in plaintext on the filesystem, and it is trivially easy to commit it by mistake. A missing .gitignore or a too-hasty git add ., and your credentials land in the Git history forever. Even after deleting the file, the previous commits remain accessible.

Environment variables are no safer. They are visible in /proc/<pid>/environ, show up in crash logs, are inherited by child processes, and remain accessible to any process running under the same user. Treating environment variables as a secure mechanism for storing secrets is a fundamental mistake.

The scale of leaks on GitHub

GitHub has automatically scanned public repositories since 2019 with its Secret Scanning program. The numbers are sobering: millions of secrets are detected every year across public repos. API tokens, AWS keys, database passwords, TLS certificates. Attackers automate the harvesting: bots monitor public commits in real time and exploit secrets within minutes of their publication.

Private repositories are not spared. A developer forking a private project into a public personal repo, a Git backup pushed to the wrong remote, or simply a change in repository visibility is enough to expose the entirety of historical secrets.

Real-world incidents

Incidents tied to secret leaks recur with predictable regularity. AWS keys hardcoded into source code have let attackers spin up hundreds of cryptocurrency mining instances, racking up six-figure bills in a matter of hours. Exposed corporate Slack tokens have granted access to entire internal conversations, including confidential channels. Production database credentials, accidentally pushed to public repos, have led to massive customer data exfiltrations.

The common denominator across all these incidents is not the sophistication of the attack. It is the absence of systemic mechanisms to prevent secrets from reaching places they should never be.

HashiCorp Vault: the industry standard

HashiCorp Vault is the benchmark for centralized secrets management. It provides a single point for storing, accessing and auditing all the secrets across your infrastructure. Vault does more than store values: it manages their full lifecycle, from creation to revocation, including automatic rotation.

Architecture: unsealing, policies and auth methods

Vault starts in a sealed state. In this state, it knows the physical location of its encrypted data but does not hold the key to decrypt it. The unsealing process requires a minimum number of decryption keys (Shamir scheme), held by different operators. No single person can unlock Vault.

# Initialize Vault with 5 keys, threshold of 3
vault operator init -key-shares=5 -key-threshold=3

# Unsealing (repeat 3 times with different keys)
vault operator unseal <key_1>
vault operator unseal <key_2>
vault operator unseal <key_3>

# Check the status
vault status

Policies define precisely who can access what. They are written in HCL (HashiCorp Configuration Language) and follow the principle of least privilege. Each policy specifies paths and the operations allowed on those paths.

# Policy for the payment application
path "secret/data/payment/*" {
  capabilities = ["read"]
}

path "database/creds/payment-readonly" {
  capabilities = ["read"]
}

# Explicit denial of write operations
path "secret/data/payment/*" {
  capabilities = ["deny"]
  # This rule takes precedence over the read capabilities
}

# Access to renew its own token
path "auth/token/renew-self" {
  capabilities = ["update"]
}

Auth methods determine how clients prove their identity to Vault. In production, the most relevant methods are: Kubernetes (pods authenticate via their ServiceAccount), AppRole (for non-Kubernetes applications), OIDC/JWT (for CI/CD pipelines), and TLS certificates (for machine-to-machine mTLS).

# Enable the Kubernetes auth method
vault auth enable kubernetes

# Configure the link with the cluster
vault write auth/kubernetes/config
  kubernetes_host="https://kubernetes.default.svc"
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create a role binding a ServiceAccount to a policy
vault write auth/kubernetes/role/payment-app
  bound_service_account_names=payment-sa
  bound_service_account_namespaces=production
  policies=payment-policy
  ttl=1h

Practical installation and configuration

In production, Vault runs in high-availability mode with a distributed storage backend. The most common setup uses Raft (the built-in consensus) or Consul as the backend. Here is a minimum viable production configuration.

# /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault-cert.pem"
  tls_key_file  = "/opt/vault/tls/vault-key.pem"
}

api_addr     = "https://vault-1.internal:8200"
cluster_addr = "https://vault-1.internal:8201"

# Telemetry for monitoring
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname         = true
}

Automatic secret rotation

Automatic rotation is the feature that sets Vault apart from a simple encrypted vault. With dynamic secrets, Vault generates credentials on the fly, with a limited lifetime. The application never knows the real database password: it asks Vault for temporary access, Vault creates a dedicated user with minimal permissions, and revokes it automatically when the TTL expires.

# Configure the secrets engine for PostgreSQL
vault secrets enable database

vault write database/config/production-pg
  plugin_name=postgresql-database-plugin
  allowed_roles="app-readonly,app-readwrite"
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/production?sslmode=require"
  username="vault_admin"
  password="initial-strong-password"

# Define a role with ephemeral credentials
vault write database/roles/app-readonly
  db_name=production-pg
  creation_statements="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";"
  revocation_statements="DROP ROLE IF EXISTS "{{name}}";"
  default_ttl="1h"
  max_ttl="24h"

# The application requests dynamic credentials
vault read database/creds/app-readonly
# Returns a unique username/password, valid for 1h

This approach eliminates three major problems: credentials shared between services, credentials that never change, and the inability to revoke a specific access without impacting other consumers.

Vault Agent for dynamic injection

Vault Agent is a sidecar process that automates authentication and the injection of secrets into applications. The application no longer needs to know the Vault API: the Agent authenticates, fetches the secrets, renders them as templated files, and handles automatic renewal.

# vault-agent-config.hcl
auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = {
      role = "payment-app"
    }
  }

  sink "file" {
    config = {
      path = "/home/vault/.vault-token"
    }
  }
}

template {
  source      = "/etc/vault-agent/templates/db-config.ctmpl"
  destination = "/app/config/database.yml"
  perms       = 0600
  command     = "pkill -HUP myapp"
}

template {
  source      = "/etc/vault-agent/templates/api-keys.ctmpl"
  destination = "/app/config/api-keys.json"
  perms       = 0600
}

The template file uses the Consul Template syntax to inject the values dynamically.

{{ with secret "database/creds/app-readonly" }}
database:
  host: db.internal
  port: 5432
  username: {{ .Data.username }}
  password: {{ .Data.password }}
  sslmode: require
{{ end }}

When the secret's TTL nears expiry, Vault Agent automatically fetches new credentials, regenerates the file, and runs the application reload command. Zero human intervention, zero downtime.

Kubernetes: External Secrets Operator and Sealed Secrets

Kubernetes brings its own challenges for secrets management. Native Secret objects are base64-encoded, not encrypted. Anyone with access to the namespace (or to etcd) can read the secrets in plaintext. Two complementary approaches solve this problem: External Secrets Operator to sync secrets from an external manager, and Sealed Secrets for GitOps-compatible encryption.

External Secrets Operator: connecting Kubernetes to secret managers

The External Secrets Operator (ESO) automatically syncs secrets from an external provider (Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) into Kubernetes Secret objects. The application in the cluster consumes a standard Secret, but the source of truth remains the external manager.

# Installation via Helm
# helm repo add external-secrets https://charts.external-secrets.io
# helm install external-secrets external-secrets/external-secrets

# ClusterSecretStore: global connection to Vault
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets-sa"

# ExternalSecret: sync a specific secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: payment-db-secret
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/payment/database
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/payment/database
        property: password

ESO periodically checks (refreshInterval) whether the secret has changed on the provider side and updates the Kubernetes Secret automatically. Combined with Vault's dynamic rotation, this creates a fully automated secrets pipeline.

Sealed Secrets: GitOps-friendly encryption

Sealed Secrets by Bitnami takes the opposite approach: instead of syncing from an external service, it lets you encrypt secrets directly in your Git manifests. A controller in the cluster holds the private decryption key. The committed manifests contain encrypted SealedSecret objects that only the cluster can decrypt.

# Install the controller in the cluster
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml

# Install kubeseal on the client side
# brew install kubeseal (macOS)

# Create a regular Secret, then seal it
kubectl create secret generic payment-api-key
  --from-literal=api-key=sk_live_abc123
  --dry-run=client -o yaml |
  kubeseal --format yaml > sealed-payment-api-key.yaml

# The resulting SealedSecret (safe to commit)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: payment-api-key
  namespace: production
spec:
  encryptedData:
    api-key: AgBy3i... (asymmetric encryption)

The major upside: SealedSecrets can be committed to Git with no risk. The asymmetric encryption guarantees that only the controller in the cluster can decrypt the values. Developers can create and modify secrets via pull request, with review and audit trail, without exposing the values in plaintext.

Comparing the approaches

The choice between ESO and Sealed Secrets depends on your context. ESO is preferable when you already have a centralized secret manager (Vault, AWS SM, GCP SM) and want a single source of truth outside the cluster. Automatic rotation is native. Sealed Secrets is a better fit for teams practicing strict GitOps who want everything in Git, including secrets (encrypted). The two approaches can be combined: ESO for critical dynamic secrets, Sealed Secrets for sensitive static configuration.

CI/CD: managing secrets in pipelines

CI/CD pipelines are a critical pressure point for secrets. They need to access image registries, deployment clusters, test databases, third-party APIs. Every secret injected into a pipeline is a secret potentially exposed in the logs, the build artifacts, or the runner's environment variables.

GitHub Actions: secrets and OIDC

GitHub Actions offers native secret storage (encrypted at rest, injected as environment variables). To go further, OIDC authentication lets you connect to Vault or cloud providers without storing long-lived credentials.

# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

permissions:
  id-token: write  # Required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # OIDC authentication to Vault
      - name: Import Secrets from Vault
        uses: hashicorp/vault-action@v3
        with:
          url: https://vault.internal:8200
          method: jwt
          role: github-deploy
          jwtGithubAudience: https://vault.internal
          secrets: |
            secret/data/deploy/production db_password | DB_PASSWORD ;
            secret/data/deploy/production api_key | API_KEY

      # The secrets are available as env vars
      - name: Deploy
        run: |
          ./scripts/deploy.sh
        env:
          DB_PASSWORD: ${{ env.DB_PASSWORD }}
          API_KEY: ${{ env.API_KEY }}

GitLab CI: variables and Vault integration

GitLab CI offers a native integration with Vault via JWTs. GitLab's CI/CD variables support environment protection and log masking, but the Vault approach remains preferable for critical secrets.

# .gitlab-ci.yml
deploy:
  stage: deploy
  image: alpine:latest
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.internal
  secrets:
    DATABASE_PASSWORD:
      vault:
        engine:
          name: kv-v2
          path: secret
        path: production/database
        field: password
      token: $VAULT_ID_TOKEN
  script:
    - ./deploy.sh

The advantage of this approach: no secret is stored in the GitLab CI variables. The pipeline authenticates against Vault with an ephemeral JWT token, fetches the secret for the duration of the job, and the token is automatically invalidated when execution ends. To learn more about GitLab CI integration, see the dedicated tutorial.

SOPS + age: file encryption for GitOps

SOPS (Secrets OPerationS) by Mozilla encrypts the values in YAML, JSON or INI files while leaving the keys in plaintext. Combined with age (the successor to PGP for file encryption), it offers a simple and auditable encryption workflow.

# Install sops and age
# brew install sops age (macOS)

# Generate an age key
age-keygen -o keys.txt
# Public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

# Create a .sops.yaml file for the configuration
cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: .enc.yaml$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
EOF

# Encrypt a secrets file
sops --encrypt secrets.yaml > secrets.enc.yaml

# The encrypted file (committable to Git)
# The keys are readable, the values are encrypted
# database:
#   password: ENC[AES256_GCM,data:abc123...,type:str]
#   host: ENC[AES256_GCM,data:def456...,type:str]

# Decrypt for use
export SOPS_AGE_KEY_FILE=./keys.txt
sops --decrypt secrets.enc.yaml

SOPS integrates natively with Flux CD and ArgoCD for automatic decryption at deploy time. The age decryption key is stored as a Kubernetes Secret in the cluster, and the GitOps controller decrypts the files on the fly.

Leak detection: scan before it is too late

Prevention is not enough. You also need to detect secrets that slip through despite the safeguards. Three tools stand out for proactive scanning of secrets in code and Git repositories.

Gitleaks: local and CI scanning

Gitleaks scans the entire Git history for secrets. It works locally (pre-commit hook) and in CI/CD pipelines to block commits containing secrets.

# Installation
# brew install gitleaks (macOS)

# Scan the entire repository history
gitleaks detect --source . --verbose

# Scan only the staged changes (pre-commit)
gitleaks protect --staged

# Integration as a pre-commit hook
cat > .pre-commit-config.yaml <<EOF
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.22.0
    hooks:
      - id: gitleaks
EOF

# Custom configuration (.gitleaks.toml)
cat > .gitleaks.toml <<EOF
[extend]
useDefault = true

[[rules]]
id = "custom-api-token"
description = "Custom API Token"
regex = '''myapp_token_[a-zA-Z0-9]{32}'''
tags = ["custom", "api"]

[allowlist]
paths = [
  '''test/fixtures/.*''',
  '''docs/examples/.*'''
]
EOF

TruffleHog: advanced detection and verification

TruffleHog goes beyond pattern matching: it actively verifies whether detected secrets are still valid by attempting an authentication (with no destructive action). This verification drastically reduces false positives.

# Scan a Git repository
trufflehog git https://github.com/org/repo --only-verified

# Scan a filesystem
trufflehog filesystem --directory /app/src --only-verified

# Scan Docker images
trufflehog docker --image myapp:latest

# CI integration (GitHub Actions)
# - name: TruffleHog Scan
#   uses: trufflesecurity/trufflehog@main
#   with:
#     extra_args: --only-verified

GitHub Advanced Security

For organizations on GitHub, Advanced Security provides secret scanning push protection. This feature blocks git push operations containing detected secrets before they even reach the repository. It covers more than 200 types of secrets from partners (AWS, Azure, GCP, Stripe, Slack, and many more).

Be sure to enable push protection in addition to passive scanning. Passive scanning detects secrets already committed (and by then it is too late). Push protection blocks them before commit.

Best practices and operational checklist

Beyond the tools, secrets management rests on organizational practices and clear policies. Here are the core principles and an actionable checklist.

The non-negotiable principles

Zero plaintext secrets, anywhere. No secret should exist in plaintext outside a secret manager or the memory of an authorized process. Not in Git, not in CI variables, not in unencrypted config files, not in Slack messages, not in Jira tickets. If a human can read a secret without a decryption tool, that is a flaw.

Systematic rotation. Every secret must have a maximum lifetime. Vault's dynamic credentials (one-hour TTL) are the ideal. For irreducible static secrets (encryption keys, third-party API tokens), set a mandatory rotation policy (90 days maximum). Automate rotation as much as possible to eliminate the human factor.

Absolute least privilege. Every service, every pipeline, every user accesses only the secrets it strictly needs. A notification service does not have access to the payment database credentials. A test pipeline does not access production secrets. Segment your secrets by environment, by service, and by sensitivity level.

Complete audit trail. Every access to a secret must be logged: who accessed which secret, when, from where. Vault provides an exhaustive audit log by default. These logs feed your SIEM and enable detection of abnormal access. If you cannot answer the question "who read this secret last week," your secrets management is incomplete.

Operational checklist

Use this checklist as a regular audit of your secrets management posture.

Storage and access:

• All production secrets are in a dedicated manager (Vault, AWS SM, GCP SM)
• No plaintext secrets in source code, config files, or CI variables
• Production .env files do not exist (secrets are injected dynamically)
• Secret access follows the principle of least privilege
• Vault policies are reviewed quarterly

Rotation and lifecycle:

• Database credentials are dynamic (TTL < 24h)
• API tokens are rotated every 90 days maximum
• Encryption keys are rotated annually
• An emergency revocation process is documented and tested
• Departing employees' secrets are revoked within 24h

Detection and prevention:

• Gitleaks or equivalent as a pre-commit hook on all repositories
• Secret scanning in every CI/CD pipeline
• Push protection enabled on GitHub/GitLab
• Periodic scanning of the entire Git history
• Automatic alerting on detection of an exposed secret

Infrastructure and monitoring:

• Vault in high availability (minimum 3 nodes)
• Regular backup of the Vault storage backend
• Vault audit logs sent to the SIEM
• Alerts on repeated authentication failures
• Vault disaster recovery test performed twice a year

Kubernetes (if applicable):

• Encryption at rest enabled for etcd
• External Secrets Operator deployed and configured
• Kubernetes RBAC limiting access to Secret objects
• Network policies isolating the Vault namespace
• Sealed Secrets or SOPS for secrets in Git

Going further

Secrets management is part of a broader security strategy. Start by hardening your Linux servers with the fundamentals. Put in place a Zero Trust architecture so that every access is verified. Secure your SSH access with keys and hardened configurations. And if you deploy with Docker, make sure your containers are not running secrets as plaintext environment variables.

Conclusion

Secrets management is not a complex technical problem. The tools exist, the patterns are documented, and the solutions are mature. Vault handles the full lifecycle. ESO and Sealed Secrets integrate with Kubernetes. SOPS encrypts files for GitOps. Gitleaks and TruffleHog detect leaks.

The real challenge is organizational: getting every team, every developer, every pipeline to adopt these practices systematically. A single .env file committed by mistake undoes months of work on secrets security. The solution is automation: pre-commit hooks, push protection, CI scanning, dynamic injection. The harder it is to do the wrong thing, the less often it happens.

Start with scanning (Gitleaks as a pre-commit hook, 15 minutes to set up). Then migrate to Vault or a cloud secret manager for production. Finally, automate rotation. Each step drastically reduces your attack surface. There is no excuse in 2026 for having plaintext secrets in your infrastructure.