How to Rotate Secrets and Credentials on a Self-Hosted App Without Breaking Production

Before you begin

• Know where the current secret lives: .env, a mounted file, the app admin UI, a database role, or a third-party console.
• Know which services consume it: web app, workers, cron jobs, reverse proxy auth, SMTP relay, queue worker, or external automation.
• Have a maintenance path ready if the app cannot tolerate partial authentication failures during the change.

This guide is about live secret rotation, not only secret storage. Existing secret-management guides explain how to keep credentials out of Git. Rotation is the next operational problem: how to replace a credential cleanly when it expires, leaks, or simply needs better hygiene.

Step 1: Inventory every dependency before you touch the secret

Write down exactly what the credential controls and where it is consumed. This sounds slow, but it is the difference between a deliberate change and a guessing game. A database password may be used by the web container, background workers, a backup job, a migration task, and an analytics importer. An SMTP password may be used by the main app and a separate notification worker. Miss one consumer and you create a partial outage.

# Example inventory notes
secret: POSTGRES_PASSWORD
used by:
- app container
- worker container
- backup job
- migration command

location:
- .env
- backup script environment

If the secret belongs to an external service, confirm whether it supports overlapping credentials, multiple API keys, or token rollouts. Dual-validity windows make rotation far safer than hard swaps.

Step 2: Choose the safest rotation pattern

The safest pattern is usually one of these:

• Dual credential window: create a new key or password while the old one still works, update consumers, verify, then revoke the old value.
• Maintenance-window swap: stop writes or pause the app briefly, rotate the credential everywhere, start in a controlled order, then test.
• Side-by-side config rollout: mount a new secret file or env source first, then restart one service at a time.

Prefer a dual-validity window whenever the system supports it. Direct cutovers are riskier because every consumer must be correct immediately.

Step 3: Prepare rollback and a test window

Before editing files, define the rollback point. That usually means keeping the old credential valid, saving the current env file, and knowing how to restart the old working configuration fast.

# Example backup of the current env file
cp .env .env.before-secret-rotation

# Example Compose view check before restarting anything
docker compose config > /tmp/compose-rendered-before-rotation.yaml

If the app has a queue, scheduled job, or background worker, decide whether to pause it first. Background tasks often surface forgotten credentials before the web UI does.

Step 4: Update secret sources cleanly

Make the source of truth consistent. If you use env files, update the value in one place and redeploy from there. If you use a mounted secret file, write the new file and confirm the path has not changed. If the database itself stores credentials for downstream services, update those records intentionally instead of mixing app config changes with data changes.

For Docker Compose stacks, the safest habit is to update the secret source, inspect the rendered config, and only then restart the services that consume it.

# Inspect what Compose will inject after the edit
docker compose config

# Restart only the service that needs the changed secret first
docker compose up -d app

For database passwords, create or alter the credential in the database first, but do not remove the old one until every consumer proves it can reconnect.

Step 5: Restart services in the right order

Restart order matters. Rotate the dependency first, then the immediate consumers, then secondary workers and cron-driven components. A common order is:

1. database, SMTP relay, Redis, or third-party credential source
2. main app container
3. background worker containers
4. cron jobs, backup scripts, and one-off automation

Do not bounce the whole stack blindly unless the application truly needs that. Smaller restarts keep the blast radius contained.

Step 6: Verify before revoking the old secret

Verification must cover real behavior, not just a green container status. Confirm the web app loads, background jobs can authenticate, email sends still work if SMTP changed, backups still connect if a database password changed, and logs do not show repeated auth failures.

# Examples of useful checks
docker compose logs --tail=100 app
docker compose logs --tail=100 worker
docker compose exec app env | rg 'SMTP|REDIS|POSTGRES'

If possible, trigger one real user-path action such as login, password reset, webhook receipt, or a queued job. Only after those checks should you revoke the old credential.

Step 7: Revoke the old credential and document the change

Once the new secret is proven, revoke or delete the old one. Then update your runbook so the next rotation does not start from memory alone. Good notes include the systems touched, the restart order that worked, and any hidden dependencies you discovered during the change.

If you use an encrypted repo or secrets manager, make sure the final committed or stored value matches what is now live. A live fix that never makes it back into the source of truth is how the next deploy resurrects the old credential.

Expected outcomes

• The app and workers authenticate successfully with the new credential.
• Logs stay free of repeated auth failures after the restart window.
• The old credential is revoked only after verification, not before.
• Your runbook reflects the real dependency map for the next rotation.

Troubleshooting

One container works but a worker fails: a secondary consumer still has the old secret. Check env files, mounted secret files, and cron-driven scripts separately.

The app starts but login or email actions fail: the primary web path may not exercise the rotated dependency. Trigger a real action that touches the changed credential.

You cannot tell whether the new value is loaded: inspect the rendered Compose config, mounted files, and container restart timestamps before changing more things.

The rotation is going sideways fast: revert to the old working credential while it is still valid, restore the prior env file, and restart the last changed consumer only. That is exactly why you kept the rollback boundary alive.

What to do next

If a risky app change is coming right after credential work, the next useful pattern is a proper write-freeze window. Continue with How to Put a Self-Hosted App Into Maintenance Mode for Safe Updates and Migrations.