A Reusable Provisioning & Deploy Agent for Self-Hosted Hetzner

📋Summary

• Problem: Running several self-hosted apps solo means provisioning, deploying, securing, and backing up multiple servers — by hand, that doesn't scale to one person.
• Solution: A small, idempotent set of scripts — one bootstrap, one deploy, one backup — that work the same across Hetzner, AWS, and GCP and converge on Docker Compose.
• Impact: New project or new server goes live in a single command; deploys are repeatable and safe; every box is firewalled, TLS-secured, and backed up nightly without manual steps.

📋Context

I run several live products on self-hosted Hetzner servers — each a full Docker Compose stack with a database, cache, object storage, and reverse proxy. Doing the server setup, deploys, TLS, and backups by hand for each one is exactly the kind of repetitive, error-prone work that should be automated. So I built the automation once and reuse it everywhere.

🎯Goals, Requirements, Constraints

Non-Goals

• A general-purpose PaaS or control plane — this is purpose-built tooling for my stacks
• Kubernetes / cluster orchestration — single-box Compose is the right scale here
• Managed services — the whole point is owning the infrastructure

🏗️Approach

Provisioning, bootstrap, and deploy are separated into composable scripts. A per-cloud launch script creates the VM and generates secrets; a single cloud-init bootstrap installs Docker and configures the firewall; a single SSH-based deploy script ships the code, starts data services, runs migrations, then brings up the app. The cloud-specific part is only the VM creation — everything after is identical, because Docker Compose is the abstraction.

Key Design Decisions

1. Decision: One shared bootstrap + deploy across three clouds; cloud-specific code only for VM creation
   Why: The launch scripts differ only in how they call hcloud / aws / gcloud and inject user-data. After the box exists, bootstrap and deploy are identical, so there's one code path to maintain and no lock-in.
   Alternatives: Per-cloud deploy logic, or an IaC tool like Terraform — more power, but more to operate for a handful of single-box apps.
2. Decision: Generate secrets with openssl rand at provision time
   Why: Each environment gets unique DB/Redis/JWT/object-storage secrets via openssl rand; they're written to an env file that's git-ignored and excluded from the deploy rsync, so secrets never touch version control.
   Alternatives: A secrets manager (Vault/SSM) is heavier than needed for this footprint.
3. Decision: Staged startup — data services and migrations before the app
   Why: Bring up Postgres + Redis, wait on pg_isready, run migrations, then start the API/worker. This eliminates the classic first-deploy crash loop where the app boots against an empty database.
   Alternatives: Start everything at once and rely on restart policies — works eventually, but noisy and slower to converge.
4. Decision: Caddy for some stacks, nginx + certbot for others
   Why: Caddy auto-provisions TLS with zero config — ideal for simpler stacks. For stacks needing fine-grained rate limiting and tuned TLS, hardened nginx with a certbot renewal sidecar gives more control.
   Alternatives: Standardize on one — but the two stacks have genuinely different needs.

⚙️Implementation

Components / Modules

• Per-cloud launch scripts: launch-hetzner.sh / launch-aws.sh / launch-gcp.sh — create the VM (Ubuntu 24.04), generate secrets via openssl rand, inject the bootstrap as cloud-init user-data, wait for SSH and cloud-init to finish, then invoke deploy.
• bootstrap-server.sh (cloud-init): Idempotent first-boot setup — installs Docker CE + Compose plugin from Docker's apt repo, enables the service, and configures UFW to allow only 22/80/443 (plus 443/udp for HTTP/3).
• deploy.sh: rsync the repo (excluding .git, node_modules, build output, and all env files), copy the prod env file, build images, start Postgres/Redis, wait for health, run migrations, then bring up the full stack. Re-runnable for updates.
• backup scripts: Nightly gzipped Postgres dump (+ media tarball where relevant), integrity-checked, with retention — uploaded to MinIO with a 30-day lifecycle rule, or pruned on disk via find -mtime +30.

Data & State

• Secrets: generated per-environment with openssl rand, written to a git-ignored env file, excluded from the deploy rsync
• Persistent Docker volumes survive rebuilds, so deploys never risk the database or media

Automation & Delivery

• One command provisions the VM and deploys the full stack
• Cloud-init bootstrap runs unattended on first boot and is safe to re-run
• Backup cron installed on first deploy (nightly), logging to /var/log

              # bootstrap-server.sh — idempotent first-boot: Docker + a default-deny-ish firewall.
if ! command -v docker &>/dev/null; then
  apt-get update -q && apt-get install -y -q ca-certificates curl gnupg
  install -m 0755 -d /etc/apt/keyrings
  curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
    | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  # ... add repo ...
  apt-get install -y -q docker-ce docker-ce-cli containerd.io docker-compose-plugin
  systemctl enable --now docker
fi
if command -v ufw &>/dev/null; then
  ufw allow 22/tcp; ufw allow 80/tcp; ufw allow 443/tcp; ufw allow 443/udp
  ufw --force enable
fi
            

🛡️Security

Controls Implemented

• UFW firewall configured at bootstrap (deny by default; allow 22/80/443 + 443/udp)
• Automatic TLS — Caddy (Let's Encrypt on first request) or nginx + a certbot renewal sidecar
• Hardened reverse proxy on nginx stacks (TLS 1.2/1.3, modern ciphers, OCSP stapling, HSTS preload, rate limiting, connection limits)
• Secrets generated per-environment and kept out of version control

⚙️Operations

📊Results

Outcomes

• Repeatability: A new project or replacement server reaches TLS-secured production in a single command, identically across three clouds.
• Safety: Staged startup and migrations-before-app removed first-deploy crash loops; persistent volumes make redeploys non-destructive.
• Security posture: Every box is firewalled and TLS-secured by default, with secrets generated per-environment and kept out of git.
• Operability: Backups run nightly with integrity checks and retention; restores and full rebuilds are documented, scripted steps.

⚖️Tradeoffs

• Bash + Compose over Terraform/Kubernetes — simpler to operate solo, but less declarative and not built for multi-node scale
• Two reverse-proxy paths (Caddy and nginx+certbot) to fit different stacks, rather than one standard

🚀Next Steps

• Add an automated rollback path to the deploy scripts (currently re-deploy a previous commit by hand)
• Collapse the per-cloud launch scripts further behind a single parameterized entrypoint