AI/CD: Production-Grade Autonomous Developer Agent (Human-in-the-Loop)

📸Snapshot

📋Context

AI/CD (Autonomous Intelligence / Continuous Delivery) is a GitHub App that acts as an autonomous developer agent. <strong>Execution Model:</strong> AI/CD is intentionally designed to replace the execution capacity typically provided by junior to mid-level engineers for well-scoped tasks. Like human engineers at those levels, it does not infer intent or guess requirements. When tickets are explicit—clear scope, constraints, and acceptance criteria—the system plans, implements, tests, validates, and submits production-ready pull requests. When tickets are vague or underspecified, the system stalls or rejects them during triage, mirroring how human execution breaks down in real teams. This behavior is a feature: it prevents silent misinterpretation, enforces engineering discipline, and shifts clarity upstream where it belongs. <strong>Why This Approach Works:</strong> In practice, poorly specified tickets cause junior and mid-level engineers to spin, over-interpret, or block. AI/CD behaves the same way—but enforces that constraint explicitly through triage and rejection rather than silent time loss. Treating underspecified work as invalid input is intentional. Silent guessing is one of the primary sources of technical debt in human teams; AI/CD makes that failure mode visible and measurable. By enforcing ticket clarity and rejecting ambiguous work, the system shifts effort upstream—forcing product and engineering leadership to define work properly once, instead of paying repeated execution cost downstream. <strong>Capability:</strong> The system handles feature development and bug fixes. It treats each issue as a first-class engineering task: planning the implementation, modifying production code, authoring tests, validating results, and submitting a pull request for review. Because only explicitly labeled, well-scoped issues entered the pipeline, feature implementation success rates were high once an issue passed triage. <strong>Ticket Quality Examples (Abstracted):</strong> • Accepted: "Add pagination to /api/jobs with cursor-based pagination; update service + handler; add unit tests for pagination edge cases." • Rejected: "Make job handling better / faster." <strong>System Design:</strong> Only issues explicitly labeled for AI/CD (e.g., ai-task) entered the pipeline; unlabeled issues were ignored. When labeled issues are opened in monitored repositories, the system: (1) Evaluates if the issue is actionable (triage), (2) Creates an implementation plan with relevant code context, (3) Generates code changes automatically, (4) Self-reviews the changes for quality/security, (5) Opens a pull request for human review. The orchestrator decomposes a task into staged sub-agent work (planning, context retrieval, implementation, review, and test authoring). For eligible issues, it typically produces a complete PR: code changes, tests, and a structured PR description aligned to repo conventions—then hands off to mandatory human review. The system is designed for complete human-in-the-loop oversight - all code must be reviewed before merge. <strong>Architecture:</strong> The monorepo includes ai-cd/ (Go backend service for webhook handler, orchestrator, AI pipeline) and ai-cd-marketing/ (Next.js admin dashboard for configuration UI). Shared infrastructure includes PostgreSQL with dual-schema architecture (public schema for backend operational data, aicd_admin schema for frontend configuration), Redis job queue (Asynq), Qdrant vector database for code embeddings, and Docker Compose for local development.

⚠️Problem

Risk if Unresolved

• Engineering Risk: Accumulating technical debt becomes unmanageable, bug backlog grows until product becomes unstable, team burnout from endless manual work on routine tasks, quality degradation due to rushed fixes.
• Business Risk: Slow velocity prevents delivering customer-requested features, competitive disadvantage due to inability to ship quickly, forced to deprioritize routine maintenance in favor of critical features (technical debt spiral).

🔒Constraints & Requirements

Non-Goals

• Not Building: Fully autonomous system that auto-merges code (too risky), support for every programming language (focus on Go, JS, Python initially), complex CI/CD pipeline integration (just create PRs, let existing CI run), native mobile apps (web dashboard sufficient), real-time collaboration features (async workflow is fine)
• Out of Scope: Replacing human engineers entirely (augmentation, not replacement), handling complex architectural decisions (focus on routine tasks), training custom LLM models (use Claude/GPT-4 via APIs)

🎯Strategy

Decision Rationale

• Go + Next.js Stack: Go for high-performance backend (webhook processing, job orchestration, LLM API calls). Next.js for modern admin dashboard with Server Components (fast, secure, type-safe).
• Dual-Schema Database: Cleanly separates Go backend data (public schema) from Next.js frontend data (aicd_admin schema). Prevents ORM collisions between database/sql and Prisma.
• 5-Stage AI Pipeline: Breaking workflow into specialized stages (triage → plan → generate → review → PR) provides quality control gates. Each stage can fail independently without corrupting downstream steps.
• Multi-Tenant from Day One: Building SaaS architecture from start ensures scalability. Row-Level Security provides database-level isolation guarantees.
• Human-in-the-Loop: Never auto-merge code. AI generates PRs, humans review and approve. Maintains quality standards, builds trust, essential for production use.

🚀Execution

Rollout & Risk Controls

• Feature Flags: AI features can be disabled per-tenant, policy engine can block all or specific repos, manual approval required for new installations
• Rate Limiting: Quota enforcement (actions per month), LLM token limits with soft/hard caps, webhook throttling to prevent abuse
• Monitoring & Alerts: Sentry for error tracking (with context propagation), Prometheus metrics for system health, Slack/Discord notifications for critical events, daily quota usage summaries
• Progressive Rollout: Started with internal use only (dogfooding), gradually added 1-5 early adopter organizations, collected feedback before broader release, human review of every PR before merge (built-in safety)
• Rollback Procedures: Database migrations have rollback scripts, feature flags for instant disable, job queue can be paused/drained, GitHub App can be uninstalled cleanly
• Cost Controls: Prompt caching to reduce LLM API calls, token counting before API calls (prevent surprises), monthly budget alerts per tenant, automatic pause if quota exceeded

🏗️Architecture

System Components

• Backend (Go Service): 11 internal packages: api (HTTP handlers for webhooks/REST), db (database models, CRUD, transactions), github (GitHub API client), llm (Claude AI client with 4 agent types), embeddings (Qdrant integration, semantic search), orchestrator (Asynq job queue, worker pool, retry logic), policy (policy engine evaluation), quota (usage tracking, enforcement), notifications (multi-channel system), monitoring (Prometheus metrics - 30+), sentry (error tracking with context). Technologies: Go 1.24.3, Anthropic Claude SDK, GitHub API v57 with OAuth2 + JWT, Asynq (Redis-backed queue), PostgreSQL with database/sql, Qdrant vector database client.
• Frontend (Next.js Dashboard): Next.js 15 with App Router and Turbopack, React 19.2.3 with Server Components (default), Client Components for interactive forms only, direct database queries via Prisma (no REST API layer for reads), Server Actions for mutations. Technologies: TypeScript 5.9 strict mode, Prisma 6.19.1 (multi-schema support), NextAuth.js 5.0.0-beta.30 (GitHub OAuth), Tailwind CSS 3.4 with custom theme, shadcn/ui components (20+ Radix primitives), React Hook Form 7.69.0 + Zod 3.25.76 validation, Recharts for analytics, pnpm 9.0.0.
• Shared Infrastructure: PostgreSQL 14+ dual-schema (public schema for backend operational data with 12 migration files and Row-Level Security policies, aicd_admin schema for frontend configuration managed by Prisma migrations). Redis 7 job queue with Asynq (5 job types, exponential backoff retry, circuit breaker). Qdrant vector database (1024-dimensional embeddings, cosine similarity search, code indexing pipeline). Docker Compose for local dev (PostgreSQL 16-alpine, Redis 7-alpine, Qdrant, Prometheus, Grafana).

Data Flows

• Ingress (GitHub Issue Created): GitHub webhook → Go backend /webhook/github endpoint → validate signature, extract issue details → check tenant exists and is active → enqueue triage job in Redis (Asynq) → return 200 OK to GitHub (fast webhook response)
• Processing Stage 1 (Triage): Worker pulls job from queue → fetch issue details from database → evaluate against policy engine (labels, paths) → call Claude API (triage agent) to assess actionability → store triage result in database → if approved enqueue planning job, else mark rejected
• Processing Stage 2 (Planning): Fetch repository code from GitHub → query Qdrant for relevant code context (semantic search on issue description) → call Claude API (planning agent) with issue + code context → parse structured implementation plan → store plan in database → enqueue code generation job
• Processing Stage 3 (Code Generation): Fetch plan from database → call Claude API (generation agent) with plan + code context → parse JSON response with file modifications → store code diff in database → enqueue review job
• Processing Stage 4 (Review): Fetch generated code diff → call Claude API (review agent) to self-assess quality/security → if issues found, can trigger regeneration (max 2 retries) → if approved, enqueue PR creation job → else mark job failed with review feedback
• Processing Stage 5 (PR Creation): Create feature branch in GitHub → commit code changes from diff → open pull request with generated description → link PR to job in database → send notifications (Slack, Discord, GitHub comment) → mark job complete
• Egress (Admin Dashboard): User logs in via NextAuth.js (GitHub OAuth) → session includes user ID, current org ID, role → Server Component fetches data from database (org-scoped queries from both public and aicd_admin schemas) → render dashboard with jobs, PRs, analytics → Client Components for interactive forms (agent config, policies) → Server Actions for mutations
• Cross-Schema Data Flow: Frontend reads tenant config from aicd_admin.agents → backend reads agent config when processing jobs (cross-schema query) → frontend displays job status from public.jobs (read-only) → backend never writes to aicd_admin schema → frontend never writes to public schema (clear ownership)

🛡️Security

Controls Implemented

• Multi-Tenant Isolation (Database): Row-Level Security (RLS) policies on all tables with tenant_id foreign keys (NOT NULL constraint), PostgreSQL session context set on every connection (SET app.current_tenant_id), audit logs for all data access. Application Level: Authorization middleware checks org membership, resource access validated against current user's org, frontend filters all queries by organizationId, backend validates tenant_id on every request. Verification: Manual penetration testing (attempted cross-tenant access - all blocked), integration tests verify RLS enforcement, audit logs reviewed for unauthorized access attempts.
• Code Generation Safety: Policy Engine (label allowlist only processes issues with specific labels like "ai-task", path filtering with glob patterns for allowed/denied file modifications, change limits for max files/lines changed per PR, approval requirements force PR review before merge). AI Pipeline Controls: Review stage self-assesses generated code for security issues, human review required before merge (no auto-merge), audit trail of every LLM call (prompt hash, response, cost), circuit breaker stops jobs after repeated failures. GitHub Safeguards: No force push capability, no auto-merge (PRs always require human approval), branch protection rules enforced, installation scope limits (only repos explicitly enabled).
• API & Cost Controls: Quota Enforcement (monthly action limits per plan - Free: 100, Pro: 1000, Enterprise: unlimited; soft quota warnings at 80% usage; hard quota blocks new jobs when exceeded; per-tenant usage tracking in database). Rate Limiting: Webhook endpoint throttled (max 100/min per installation), LLM API calls counted and logged, Prometheus metrics track API usage, alerts on abnormal usage patterns. Cost Monitoring: Token counting before API calls (estimate cost), budget alerts per tenant (monthly spend), prompt caching reduces duplicate calls, automatic job pause if budget exceeded.
• Token & Credential Security: GitHub Installation Tokens (short-lived 60 minutes max, generated on-demand per job, never persisted in database, scoped to single installation). User API Keys BYOK (encrypted at rest with AES-256-GCM, ENCRYPTION_SECRET environment variable, decrypted only when needed for LLM calls, never logged or exposed in errors). Database Credentials: Environment variables only (never hardcoded), different credentials for dev/prod, connection pooling with max limits, no credentials in logs or Sentry reports.
• Audit & Observability: LLM Call Logging (every Claude API call recorded in llm_calls table with tenant_id, agent_type, provider, model, tokens_input, tokens_output, cost_usd, prompt_hash SHA-256 not full prompt for privacy). Policy Audit Trail: All policy evaluation results logged, rejected issues include rejection reason, failed policy checks tracked with details, admin UI shows policy decision history. Job Execution Audit: State transitions logged with timestamps, error messages captured (sanitized of secrets), correlation IDs link related logs, Sentry context includes tenant_id (never crosses tenant boundaries).

⚙️Operations

📊Results

💡Lessons Learned

📦Artifacts

• GitHub App (Private)
• Admin Dashboard (Private)
• Architecture Documentation (Private)
• User Guide (Private)

Selected Technical Details

// Multi-Stage AI Pipeline State Machine
type JobStatus string

const (
    JobStatusPending     JobStatus = "pending"
    JobStatusTriaging    JobStatus = "triaging"
    JobStatusPlanning    JobStatus = "planning"
    JobStatusGenerating  JobStatus = "generating"
    JobStatusReviewing   JobStatus = "reviewing"
    JobStatusCreatingPR  JobStatus = "creating_pr"
    JobStatusCompleted   JobStatus = "completed"
    JobStatusFailed      JobStatus = "failed"
    JobStatusRejected    JobStatus = "rejected"
)

// Triage worker (stage 1) - Quality gate before expensive operations
func (w *TriageWorker) Process(ctx context.Context, task *asynq.Task) error {
    var payload TriagePayload
    if err := json.Unmarshal(task.Payload(), &payload); err != nil {
        return fmt.Errorf("unmarshal payload: %w", err)
    }

    // Update job status to "triaging"
    if err := w.db.UpdateJobStatus(ctx, payload.JobID, JobStatusTriaging); err != nil {
        return err
    }

    // Evaluate against policy engine (labels, paths, change limits)
    decision, err := w.policy.Evaluate(ctx, issue)
    if err != nil {
        return err
    }

    if !decision.Allowed {
        // Policy rejected - mark job as rejected (fail early, no LLM cost)
        w.db.UpdateJobStatus(ctx, payload.JobID, JobStatusRejected)
        w.db.UpdateIssueTriageStatus(ctx, issue.ID, "rejected", decision.Reason)
        return nil // Not an error, just rejected
    }

    // Call Claude AI for triage
    triageResult, err := w.llm.Triage(ctx, issue)
    if err != nil {
        return fmt.Errorf("llm triage: %w", err)
    }

    if !triageResult.IsActionable {
        // AI rejected - mark as rejected
        w.db.UpdateJobStatus(ctx, payload.JobID, JobStatusRejected)
        w.db.UpdateIssueTriageStatus(ctx, issue.ID, "rejected", triageResult.Reason)
        return nil
    }

    // Approved - enqueue planning job
    w.db.UpdateIssueTriageStatus(ctx, issue.ID, "approved", triageResult.Reason)
    planningTask := asynq.NewTask("planning", payload)
    w.queue.Enqueue(ctx, planningTask)

    return nil
}

// Row-Level Security for Multi-Tenant Isolation
// Database-level enforcement prevents application bugs from leaking data
CREATE POLICY tenant_isolation ON issues
  USING (tenant_id = current_setting('app.current_tenant_id')::int);

// Every database connection sets tenant context
func (db *DB) SetTenantContext(ctx context.Context, tenantID int) error {
    query := fmt.Sprintf("SET app.current_tenant_id = %d", tenantID)
    _, err := db.pool.Exec(ctx, query)
    return err
}