Health-Gated, Rollback-Safe Deploys for a Self-Hosted Compose Stack

📋Summary

• Problem: Deploying a multi-service stack by hand is slow and risky — easy to forget a backup, apply a migration at the wrong time, or take the whole app down at once.
• Solution: A scripted, health-gated deploy that backs up first, migrates, restarts services sequentially behind health checks, and reloads the proxy — with rollback documented and tested.
• Impact: Repeatable, low-anxiety deploys on a single self-hosted box; failed health checks stop the rollout, and a known-good rollback is one backup-restore + git reset away.

📋Context

My self-hosted apps run as Docker Compose stacks (Go API, Next.js frontend, PostgreSQL, Redis, plus a reverse proxy and, on the heavier stack, Qdrant + Prometheus + Grafana) on Hetzner. Compose's default "up -d" recreates containers, so the honest goal isn't literal zero downtime — it's a deploy that's safe, sequenced, and trivially reversible, run by one operator.

Symptoms / Failure Modes

• Hand-run deploys skip the pre-deploy backup exactly when you need it most
• Migrations applied before the database is ready cause crash loops
• Restarting everything at once turns a small bug into a full outage with no clean rollback

🎯Goals, Requirements, Constraints

Non-Goals

• Kubernetes / service mesh / blue-green infrastructure — wrong scale for a single-box app
• True zero-downtime container swaps (would need a second environment or rolling proxy cutover)

🏗️Approach

A single deploy script orchestrates the rollout. It validates the environment and warns on a dirty git tree, takes a database backup (abort the deploy if it fails), rebuilds images, runs migrations, then restarts services sequentially — backend first and only proceeding once its health endpoint responds, then the frontend, then a reverse-proxy reload. Flags (--skip-backup, --skip-migrations) exist for controlled cases. The companion stack uses the same staged pattern — data services and migrations before the app — to avoid first-boot crash loops.

Key Design Decisions

1. Decision: Back up the database before every deploy, and abort if the backup fails
   Why: The cheapest insurance against a bad migration is a fresh, verified dump taken seconds earlier. If it can't be taken, the deploy shouldn't proceed.
   Alternatives: Periodic-only backups — leaves a gap exactly around the riskiest moment.
2. Decision: Sequential, health-gated restarts
   Why: Restart the backend and wait for its /health endpoint before moving on; a failing health check stops the rollout instead of cascading. Then the frontend, then reload the proxy.
   Alternatives: Restart everything at once — faster, but a single bad image takes the whole app down with no checkpoint.
3. Decision: Rollback via backup-restore + git reset to the previous commit
   Why: The deploy records the current commit, so reverting is straightforward — stop, restore the pre-deploy dump, git reset --hard to the prior commit, rebuild, restart. Simple and reliable.
   Alternatives: A second (blue-green) environment would enable instant cutover but doubles resources and ops for a single-box app.

⚙️Implementation

Components / Modules

• Deploy script: Pre-flight checks, pre-deploy DB backup, image build, migrations, sequential health-gated restarts, reverse-proxy reload. Re-runnable, with --skip-backup / --skip-migrations escape hatches.
• Backup-before-deploy: Calls the nightly backup script inline; a failed backup aborts the deploy.
• Health gates: Backend restart waits on its /health endpoint; the frontend on its root; the rollout halts if a check fails.
• Reverse proxy: nginx (with a certbot renewal sidecar) or Caddy (automatic TLS) fronts the stack and is reloaded after restarts.

Automation & Delivery

• Single command runs the full sequence; CI can invoke it on merge
• Pre-deploy database backup runs automatically (gzipped, integrity-checked, retained)
• Health checks gate progression; failures halt the rollout

              # deploy.sh — backup first (abort on failure), then build, migrate, restart behind health checks.
if [ "$SKIP_BACKUP" != true ]; then
  bash "$BACKUP_SCRIPT" || error "Database backup failed. Aborting deployment."
fi
docker compose -f docker-compose.prod.yml build --no-cache backend frontend
[ "$SKIP_MIGRATIONS" = true ] || bash "$INIT_DB"
# restart backend, wait for health, then frontend, then reload the proxy
docker compose -f docker-compose.prod.yml up -d backend
curl -fsS http://localhost:8080/health
docker compose -f docker-compose.prod.yml up -d frontend
            

🛡️Security

Controls Implemented

• Reverse proxy enforces TLS and security headers; app services stay on the internal network
• Secrets live in a git-ignored env file, excluded from the deploy copy

⚙️Operations

📊Results

Outcomes

• Safety: Every deploy is preceded by a verified database backup, so the worst case is a quick restore to seconds earlier.
• Reliability: Sequential health-gated restarts stop a bad build before it takes the whole app down.
• Operability: Deploys and rollbacks are single, documented commands an operator can run with confidence.

⚖️Tradeoffs

• Not literal zero-downtime: compose up -d recreates containers, so there's a brief restart window per service
• Single-box simplicity over blue-green/canary — less infrastructure, but no instant cutover

🚀Next Steps

• Add an automated rollback command (currently a documented manual sequence)
• Optional proxy-level cutover (start new container, switch upstream, drain old) for true zero-downtime on the busier stack