Infrastructure for an Autonomous Agent Platform

📋Summary

• Problem: Running untrusted, AI-generated code for many tenants needs strong isolation, reliable async processing, per-tenant data separation, and deep visibility into cost and failure — all without a sprawling cloud bill.
• Solution: Ephemeral hardened containers per job, a Redis-backed queue with retries and a dead-letter path, Qdrant for tenant-scoped code search, dual-schema Postgres with RLS, and 30+ Prometheus metrics feeding Grafana plus Sentry tracing.
• Impact: A complete platform that runs on a single mid-size server for beta, self-hosts from one docker-compose file, and surfaces per-tenant job, token, and cost metrics in real time.
• Key Decisions: Docker sandboxes with RLS for isolation, Redis/Asynq over a workflow engine for MVP simplicity, dual-schema Postgres for clean backend/frontend ownership, Qdrant for payload-filtered tenant search.

📋Context

This is the infrastructure beneath an autonomous developer agent that turns GitHub issues into pull requests. The interesting engineering isn't only the agents — it's the platform that runs them safely for many tenants: isolating execution of AI-generated code, processing jobs reliably, keeping each tenant's data and embeddings separate, and making cost and failure observable. This writeup focuses on that platform layer rather than the agent product.

Symptoms / Failure Modes

• AI-generated code can't run on shared infrastructure without strong isolation
• Long, multi-stage jobs need reliable retries and a place for permanent failures to land
• Per-tenant code context must never bleed across tenants in vector search
• LLM spend is invisible without explicit per-call token and cost accounting

🎯Goals, Requirements, Constraints

Non-Goals

• A fully managed multi-region control plane (single-server beta for now)
• Firecracker/microVM isolation (a documented future upgrade from Docker)
• Autoscaling orchestration (stateless workers, but Kubernetes is later)

Acceptance Criteria

• Each job executes in an ephemeral container with resource limits and dropped privileges
• Failed jobs retry with backoff and land in a dead-letter queue after exhaustion
• Every tenant-scoped query and embedding search is filtered by tenant
• Per-tenant job, token, and cost metrics are visible in Grafana

🏗️Approach

A GitHub webhook enqueues a job into Redis; a worker pool pulls jobs through a four-stage agent pipeline (triage, planning, code generation, review) and opens a pull request for human review. Each stage that executes code does so in a fresh Docker container — resource-limited, read-only root filesystem, dropped capabilities, non-privileged user, restricted egress — destroyed when the job ends. Postgres uses two schemas (backend-owned and frontend-owned) joined through a tenant/organization mapping, with RLS policies filtering tenant data. Qdrant holds code embeddings filtered by tenant on every search. Sentry, Prometheus, and Grafana provide tracing, metrics, and dashboards.

Key Design Decisions

1. Decision: Ephemeral Docker sandboxes for execution
   Why: AI-generated code is untrusted; fresh, resource-capped containers with dropped capabilities and restricted egress contain it, and discarding them after each job leaves no persistent state to compromise.
   Alternatives: A VM per job is far slower and costlier; running on shared workers risks cross-job contamination. Firecracker is noted as a future upgrade for high-security tenants.
2. Decision: Redis + Asynq for the job queue
   Why: A single Redis container gives reliable persistence, exponential-backoff retries, a dead-letter queue, and priority lanes with minimal operational surface for an MVP.
   Alternatives: A full workflow engine (e.g. Temporal) was rejected as overkill for a linear pipeline; the tradeoff is that Redis becomes a component to make highly available later.
3. Decision: Dual-schema Postgres with row-level security
   Why: A backend-owned schema and a frontend-owned (Prisma) schema give each side clean ownership and independent migrations, while RLS enforces tenant isolation in the kernel rather than in every query.
   Alternatives: A single shared schema invites ORM collisions and migration conflicts; a database per tenant is heavier to operate at this stage.
4. Decision: Qdrant with per-tenant payload filtering
   Why: Code-context search must never surface another tenant's code; filtering embeddings by tenant_id on every query keeps retrieval isolated with low latency.
   Alternatives: A hosted vector service adds cost and lock-in; storing embeddings in Postgres alone gives weaker semantic search ergonomics.

⚙️Implementation

Components / Modules

• Webhook + enqueue: Validates the GitHub App installation and repository, persists the issue, and enqueues a job onto the Redis queue.
• Worker pool + agent pipeline: Pulls jobs and runs the four-stage triage → planning → code-gen → review pipeline, logging each LLM call's tokens and cost.
• Ephemeral executor: Spawns a hardened, resource-limited Docker container per job (read-only root, dropped capabilities, non-privileged user, restricted egress) and tears it down afterward.
• Data plane: Dual-schema PostgreSQL 16 with RLS for tenant data and a Qdrant vector store for tenant-scoped code embeddings.
• Observability stack: Sentry for error tracing, Prometheus for 30+ custom metrics, and a pre-built Grafana dashboard with alerting on failure rate, queue depth, and cost.

Data & State

• PostgreSQL 16 with two schemas: a backend-owned schema (tenants, repositories, jobs, pull_requests, llm_calls) and a frontend-owned schema (users, organizations, agents, policies, api_keys).
• RLS policies filter tenant-scoped tables by a per-request app.current_tenant_id setting.
• Qdrant collections store code embeddings with tenant_id in the payload, filtered on every search.
• BYOK provider keys stored encrypted (AES-256-GCM) in the frontend schema.

Automation & Delivery

• GitHub App webhook drives job creation end-to-end
• Asynq provides retries with exponential backoff and an automatic dead-letter queue
• One docker-compose.prod.yml brings up all eight services; init and backup scripts handle migrations and snapshots

              // Each job runs in a throwaway, hardened container.
hostCfg := &container.HostConfig{
    Resources: container.Resources{
        Memory:   512 * 1024 * 1024, // 512MB
        NanoCPUs: 1_000_000_000,     // 1 CPU
    },
    ReadonlyRootfs: true,
    CapDrop:        []string{"ALL"},
    AutoRemove:     true, // destroyed when the job ends
}
            

Notable Challenges

• Isolating untrusted execution without a microVM — handled with hardened, ephemeral Docker containers today, with Firecracker flagged as the next step for high-security tenants.
• Keeping two schemas coherent — the backend reads across into the frontend schema for org/policy lookups via a junction table, avoiding ORM collisions while allowing cross-schema joins.
• Making LLM spend legible — every model call records provider, model, tokens, and estimated USD into llm_calls, which rolls up into Grafana cost panels and alerts.

🛡️Security

Controls Implemented

• Ephemeral containers — read-only root, dropped capabilities, non-privileged user, egress allowlist
• Row-level security on tenant-scoped tables plus mandatory tenant filters in application queries
• Short-lived, per-installation GitHub tokens with automatic refresh
• Encrypted storage of bring-your-own-key provider credentials

⚙️Operations

📊Results

Outcomes

• Isolation: Untrusted code runs only in disposable, hardened containers, and tenant data is separated by RLS in the database and payload filters in the vector store.
• Reliability: The Redis/Asynq queue gives retries with backoff and a dead-letter path, so transient failures recover and permanent ones are quarantined for review.
• Visibility: 30+ metrics and a Grafana dashboard make job throughput, token usage, and estimated cost observable per tenant in real time.

⚖️Tradeoffs

• Docker isolation is strong but not a microVM; Firecracker is the planned upgrade for the highest-security tenants.
• A single Redis instance is a potential bottleneck/SPOF that managed Redis or replication will address.
• The dual-schema design needs hand-written cross-schema queries since the backend schema isn't behind the ORM.

🚀Next Steps

• Introduce Firecracker microVMs for high-security tenant execution
• Move to managed/replicated Redis and Postgres for HA beyond single-server beta
• Add a Kubernetes deployment path for horizontal worker scaling