Building a Production-Ready Loan Origination System Integration

📸Snapshot

📋Context

In the automotive financing industry, dealerships work with multiple lenders to secure loans for their customers. The traditional process involves manual data entry into lender portals, phone calls for status updates, and fragmented communication - creating friction, delays, and error risk. System Environment: DealerClick management system built with Go + PostgreSQL, 10 pilot dealerships processing ~1,000 applications/month, appTRAKER LOS integration partner with REST API, Hetzner Cloud deployment (Ubuntu 22.04 LTS, single PostgreSQL instance). What Triggered This Work: Manual loan submission was #1 pain point in user interviews, each application took 15-20 minutes at $25/hour labor rate ($6.25 cost per app), ~5% error rate causing delays/rejections, competitive pressure from rival platforms with automated submission, revenue impact (faster approvals = more deals closed). Technical Challenge: Modern Loan Origination Systems offer APIs with integration challenges - complex nested data structures with 50+ required fields, asynchronous callback systems with variable schemas based on loan stage, two-stage webhook pattern requiring careful state management, vendor API changes without notice, PII handling compliance (GLBA, FACTA, state regulations).

⚠️Problem

Risk if Unresolved

• Business Risk - Revenue Impact: Unable to scale dealership operations, limiting growth
• Business Risk - Customer Churn: Poor experience drives customers to competitors with faster approvals
• Business Risk - Labor Costs: $6.25 per application in manual entry (15-20 min @ $25/hr) × 1,000 apps/month = $6,250/month waste
• Technical Risk - Scalability: Manual process doesn't scale with business growth
• Technical Risk - Data Quality: Error rate causes deal fallout and lender relationship damage
• Technical Risk - Compliance: No audit trail for regulatory requirements (GLBA, FACTA)
• Regulatory Risk: GLBA (Gramm-Leach-Bliley Act) requires PII protection throughout pipeline, FACTA compliance for adverse action notices, state-specific lending regulations across 50 states, audit requirements for data retention and traceability

🔒Constraints & Requirements

Non-Goals

• Multi-lender support: Single vendor (appTRAKER) for MVP, not 5+ lenders
• Real-time credit score pulls: Delegated to lender's existing process
• Automated stipulation uploads: Deferred to Phase 2 (document management)
• Mobile-first interface: Desktop dealers prioritized, mobile later
• Building custom LOS: Use existing vendor, don't reinvent loan origination
• Perfect schema normalization: JSONB flexibility > referential integrity for vendor data

🎯Strategy

Decision Rationale

• JSONB flexibility vs. referential integrity: Tradeoff - Lose DB-level foreign key constraints. Why acceptable: Vendor schema volatility outweighs benefits of strict validation. Mitigation: Application-level validation + raw_callback_payload for recovery.
• Soft validation vs. strict validation: Tradeoff - Accept malformed data, manual review required. Why acceptable: Zero data loss > perfect parsing; prevents retry storms. Mitigation: Comprehensive logging, manual review queue, alerting on parse failures.
• Synchronous submission vs. async queue: Tradeoff - Frontend waits 2-3 minutes for vendor response. Why acceptable: Simpler implementation, acceptable latency for dealers, immediate error feedback. Future: Can add async queue if volume exceeds 100 apps/hour per dealer.
• Risk reduction: Incremental rollout (pilot with 1 dealer → 3 dealers → 10 dealers over 4 weeks), feature flags (could disable submission if vendor API had issues), comprehensive monitoring (Sentry for errors, logging for all callbacks), backup plan (manual submission still available as fallback), runbook preparation (documented common issues before launch)

🚀Execution

Rollout & Risk Controls

• Progressive Delivery: Week 1 - Single dealer pilot, 100% monitoring, daily check-ins. Week 2 - Three dealers, callback success rate validation. Week 3-4 - Ten dealers, performance optimization, cost monitoring.
• Safety Mechanisms: Daily PostgreSQL backups with 30-day retention, .down.sql migration scripts tested in staging, feature flag (could disable submission if vendor API issues detected), manual fallback (dealers could still use old manual process if needed)
• Operational Readiness: Runbook created for vendor API downtime, callback parsing failures, database issues. Alerts configured for callback success rate <95%, vendor latency >5s, DB pool >80%. Dashboard created for loan application flow, webhook health, database performance. Team trained with 373-line frontend integration guide. Support docs updated with dealer FAQ on new loan submission process.

🏗️Architecture

System Components

• routes/loan_application.go (712 lines): HTTP handlers for loan submission and webhook callbacks. Interfaces: POST /loan-application (dealer submits loan, proxies to vendor), POST /loan-application/status (Stage 1 callback - app_no linking), POST /loan-application/{lenderName} (Stage 2+ callbacks - status updates). Boundaries: Owns HTTP layer, delegates to database layer. Scaling: Stateless, can horizontally scale with load balancer.
• db/submitted_app.go (246 lines): Database operations with pgx connection pooling. Interfaces: CreateSubmittedApp (atomic insert with JSONB marshaling), UpdateSubmittedAppNo (links vendor ID from first callback), UpdateSubmittedAppCallbackData (upserts callback data into JSONB columns). Boundaries: Owns data access, no business logic. Scaling: Connection pool (max 25 connections), prepared statements.
• models/submitted_app_models.go (80 lines): Go structs for type safety, TypeScript interfaces for frontend. Interfaces: SubmittedApp, AppTrackerStatusCallback, LoanRequestWrapper. Boundaries: Shared types across backend and frontend.
• PostgreSQL 14+ Database: Primary database with JSONB support. Schema: Hybrid approach (typed columns for IDs/status, JSONB for flexible data). Indexes: idx_submitted_apps_app_no, idx_submitted_apps_dealer_id. Retention: Indefinite (regulatory requirement), consider archiving >2 years to S3 Glacier. Growth: ~500 MB/month with 1,000 apps/month, projected 30 GB over 5 years.
• Frontend (React): Loan application form with validation, real-time status display, lender communication tracking. Integration: POST /loan-application, GET /submitted-apps?dealer_id={id}
• appTRAKER LOS (External Vendor): POST https://apptkgold51-services.apptraker.com/..., returns app_id + app_no + status, sends callbacks on status changes

Data Flows

• Loan Submission Flow (Synchronous): Dealer (Frontend) → POST /loan-application (Go API) → Validate payload (50+ fields) → Proxy to appTRAKER API (30s timeout) → Receive app_id from vendor → INSERT into submitted_apps (JSONB) → Return app_id to frontend (2-3 min total)
• Stage 1 Callback Flow (30-120s after submission): appTRAKER → POST /loan-application/status → Validate API key + payload → Extract app_id and app_no → UPDATE submitted_apps SET app_no WHERE submitted_app_id → Return 200 OK
• Stage 2+ Callback Flow (ongoing status updates): appTRAKER → POST /loan-application/{lenderName} → Validate lender allowlist + API key → Parse callback payload (soft validation) → Marshal nested JSON to JSONB columns → UPDATE submitted_apps WHERE app_no (text cast) → Store raw_callback_payload for audit → Return 200 OK (even if parse fails)
• Security Boundaries: Frontend ↔ Backend (JWT authentication for dealer sessions), Backend ↔ Database (localhost connection, no external exposure), Backend ↔ Vendor API (Mutual TLS, API key in X-API-KEY header), Vendor ↔ Backend Webhooks (API key validation + lender allowlist)
• PII Handling: In Transit (TLS 1.2+ for all external communication), At Rest (PostgreSQL encryption at rest via Hetzner Cloud volume encryption), In Logs (PII redacted in error logs - SSN masked, addresses omitted), In Code (No hardcoded credentials, secrets in environment variables)

🛡️Security

Controls Implemented

• Identity & Access Management: Database user has INSERT, UPDATE, SELECT only on submitted_apps table (no DELETE or DROP permissions to prevent accidental data loss), API key stored in .env file (git-ignored, never in code), separate API keys for sandbox vs. production environments, quarterly secret rotation (vendor requirement)
• Network Security: Backend API only accepts HTTPS (TLS 1.2+), database only accessible via localhost (no public exposure), vendor API calls use mutual TLS (client certificate validation), firewall rules restrict inbound traffic to port 443 only
• Secrets Management: Vendor API keys in environment variables (.env git-ignored), production secrets managed via Hetzner Cloud secrets manager, no hardcoded credentials in codebase (enforced by pre-commit hook), secrets rotated quarterly
• Lender Allowlist: approvedLenders array validates lenderName from webhook path parameter, unauthorized lenders rejected with 403 Forbidden, logged for security monitoring
• Secure SDLC: gosec static analysis in CI/CD pipeline, Dependabot alerts + weekly govulncheck for dependency scanning, database migrations reviewed for data loss risk, critical vulnerabilities patched within 48 hours
• Audit & Compliance: All webhook callbacks logged with timestamp, lender, app_no. raw_callback_payload stored for every callback (compliance requirement). Error logs sanitize PII (SSN redacted, addresses omitted). Logs retained for 7 years (regulatory requirement).

⚙️Operations

📊Results

💡Lessons Learned

📦Artifacts

• Live System
• routes/loan_application.go
• db/submitted_app.go
• LOAN_STATUS_CALLBACK_UI_GUIDE.md
• Database Migrations

Selected Technical Details

// Defensive JSON Parsing Pattern
// routes/loan_application.go:608-642
// Handles malformed callbacks gracefully
var payload AppTrackerStatusCallback
raw, _ := io.ReadAll(r.Body)

if err := json.NewDecoder(bytes.NewReader(raw)).Decode(&payload); err != nil {
    log.Printf("ERROR: Failed to decode appTRAKER callback: %v", err)
    log.Printf("WARNING: Accepting callback despite parse error")

    // Attempt partial extraction for database lookup
    var partialPayload map[string]interface{}
    if err := json.Unmarshal(raw, &partialPayload); err == nil {
        if app, ok := partialPayload["Application"].(map[string]interface{}); ok {
            // Handle both float64 and int (JSON number ambiguity)
            var appnoStr string
            if appno, ok := app["Appno"].(float64); ok && appno > 0 {
                appnoStr = fmt.Sprintf("%.0f", appno)
            } else if appno, ok := app["Appno"].(int); ok && appno > 0 {
                appnoStr = fmt.Sprintf("%d", appno)
            }

            if appnoStr != "" {
                // Store raw payload only for manual review
                db.UpdateSubmittedAppCallbackData(ctx, appnoStr, raw, nil)
            }
        }
    }

    // Return 200 OK to prevent vendor retry storms
    w.WriteHeader(http.StatusOK)
    return
}

// Dual Identifier Lookup (UUID + INTEGER)
// db/submitted_app.go:215-217
WHERE submitted_app_id::text = $12
   OR app_no::text = $12

// Status Precedence Logic
// routes/loan_application.go:656-663
newStatus := payload.Application.DecisionStatus
if payload.Application.FundingStatus != "" &&
   payload.Application.FundingStatus != "Undefined" {
    newStatus = payload.Application.FundingStatus
    log.Printf("INFO: Using FundingStatus: %s", newStatus)
} else {
    log.Printf("INFO: Using DecisionStatus: %s", newStatus)
}