Multi-Source Media Ingestion: YouTube, RSS, and File Metadata

📋Summary

• Problem: Pulling media from heterogeneous sources — APIs, feeds, and uploaded files in many formats — usually means bespoke, brittle code per source and no unified way to browse across them.
• Solution: A consistent per-domain ingestion pattern that normalizes YouTube, RSS/Atom, audio, audiobooks, ebooks, and podcasts into shared types, with resilient parsing and SSRF-safe fetching.
• Impact: Five media domains ingest through one repeatable pattern; users browse and search across sources, work offline by default, and opt into server sync — with no recommendation algorithm anywhere.
• Key Decisions: LocalStorage-first with optional sync, normalization across sources, in-process metadata parsing (no external services), and an SSRF allowlist on all outbound fetches.

📋Context

The platform is a self-hosted media app built around a deliberate constraint — no recommendation algorithm; users curate their own feeds. That makes ingestion the heart of the system: it has to pull from YouTube, arbitrary RSS/Atom feeds, uploaded audio and ebooks, and podcast directories, then present them through a consistent interface. The engineering challenge is integration breadth — many sources, many formats, all normalized and all parsed defensively.

Symptoms / Failure Modes

• Each media source has its own API, feed dialect, or file format
• RSS in the wild is inconsistent (RSS 2.0 vs Atom, varied thumbnail conventions)
• Uploaded media carries unreliable or missing metadata
• Fetching user-supplied feed URLs is a server-side request forgery risk

🎯Goals, Requirements, Constraints

Non-Goals

• Any recommendation or ranking algorithm (excluded by design)
• A mandatory account or third-party identity provider
• Server-side transcoding of uploaded media (metadata and streaming only)

Acceptance Criteria

• YouTube and RSS video sources merge into one filterable list
• Audio, audiobook, and ebook uploads yield titles, art, and chapters where present
• Podcasts resolve from the Apple directory to an open RSS feed
• Malformed feeds or files degrade gracefully instead of failing the request

🏗️Approach

Each of the five domains follows the same shape — domain types, a versioned localStorage store, an optional API proxy to Postgres, a shared context provider, and an upload/ingest service that validates MIME and size. Video ingestion merges the YouTube Data API (channel resolution plus recent uploads, Shorts filtered out) with a custom RSS/Atom parser into one `VideoItem` type. Audio and audiobooks extract ID3/iTunes metadata in-process; ebooks parse EPUB as a ZIP via its OPF manifest; podcasts resolve through the Apple lookup API to a standard feed. Every outbound fetch passes an SSRF allowlist, and every parser has a fallback path.

Key Design Decisions

1. Decision: LocalStorage-first with optional Postgres sync
   Why: The app works offline and without an account by default; server sync is a transparent opt-in, which keeps the read-heavy common case fast and reduces server load.
   Alternatives: A server-first model would require accounts and infrastructure for what is fundamentally a personal, local-first experience.
2. Decision: Normalize heterogeneous sources into shared types
   Why: Mapping YouTube and RSS into one VideoItem (and similar across domains) decouples the UI from source specifics, so browsing, searching, and filtering work uniformly.
   Alternatives: Source-specific UIs would multiply code and fracture the browse/search experience.
3. Decision: In-process metadata parsing (music-metadata, JSZip, fast-xml-parser)
   Why: Pure-JS parsing pulls ID3 tags, M4B chapters, and EPUB structure with no external metadata API — no per-lookup cost, no third-party dependency, works fully self-hosted.
   Alternatives: External metadata services add latency, cost, and a network dependency the self-hosted goal forbids.
4. Decision: SSRF allowlist on all outbound fetches
   Why: Users supply feed URLs, so requests must reject private/internal ranges and require HTTPS to prevent the server from being turned into a proxy into its own network.
   Alternatives: Fetching user URLs unguarded is a classic SSRF foothold.

⚙️Implementation

Components / Modules

• Video ingestion: YouTube Data API channel resolution and recent uploads (batched, Shorts filtered) merged with a custom RSS/Atom parser into a single VideoItem type.
• Audio & audiobook ingestion: Streaming uploads (busboy) with music-metadata extracting ID3 tags and M4B chapters, falling back to filename and a single chapter when metadata is absent.
• Ebook ingestion: EPUB parsed as a ZIP via JSZip — OPF manifest, spine, HTML-stripped chapters, cover detection — with TXT chapter-splitting and PDF stored for later extraction.
• Podcast ingestion: Apple Podcasts lookup resolving an iTunes ID to an open RSS feed, with an iTunes-aware parser for duration, episode, and artwork.
• Import/export: Subscriptions importable from Google Takeout CSV, OPML, and a native JSON format for portability.

Data & State

• Five domains (videos, music, audiobooks, books, podcasts), each with versioned localStorage keys and an optional Postgres sync layer.
• Unified per-domain types (e.g. VideoItem across YouTube and RSS) so the UI is source-agnostic.
• Subscriptions deduplicated by stable keys (yt:channelId or rss:url).
• Custom JWT auth (jose, HS256) over httpOnly cookies with bcrypt — no third-party auth service.

Automation & Delivery

• Batched fetching (up to 50 channel IDs and 25 RSS URLs per request)
• HTTP Range request support so audio/video can seek without re-downloading
• Vitest unit tests plus Playwright end-to-end coverage

              // Outbound feed fetches pass an SSRF allowlist before any request.
function assertSafeFeedUrl(raw: string): URL {
  const url = new URL(raw);
  if (url.protocol !== "https:") throw new Error("HTTPS required");
  const host = url.hostname;
  if (
    host === "localhost" || host.endsWith(".local") ||
    /^(127\.|10\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.)/.test(host)
  ) throw new Error("Private network blocked");
  return url;
}
            

Notable Challenges

• RSS in the wild is inconsistent — handled with conditional RSS 2.0 vs Atom parsing and flexible thumbnail extraction across media:thumbnail, enclosure, and itunes:image.
• Filtering YouTube Shorts — duration-based exclusion for playlist items plus explicit /shorts/ path detection in Atom feeds.
• EPUB OPF paths and M4B chapter atoms vary by producer — solved with dynamic OPF resolution, multiple cover-path fallbacks, and a single-chapter fallback when metadata is missing.
• Parsing failures shouldn't break ingestion — every parser is wrapped so malformed input degrades to sane defaults (filename title, 'Unknown Artist', empty list) instead of erroring.

🛡️Security

Controls Implemented

• SSRF allowlist (private-range blocklist + HTTPS enforcement) on all outbound fetches
• Streaming uploads via busboy to avoid buffering large files in memory
• MIME-type and file-size validation per media domain
• Self-hosted custom JWT auth with httpOnly cookies and bcrypt

⚙️Operations

📊Results

Outcomes

• Breadth: Five media domains — video, music, audiobooks, books, podcasts — ingest through one repeatable per-domain pattern.
• Resilience: Defensive parsing means inconsistent feeds and missing file metadata degrade gracefully instead of failing ingestion.
• Privacy & control: Local-first storage, optional sync, self-hosted auth, and no recommendation algorithm keep users in control of their feeds.

⚖️Tradeoffs

• PDF ingestion currently stores the file but defers content extraction, unlike EPUB which is fully parsed.
• LocalStorage-first simplifies the common case but pushes conflict handling to the optional sync layer.
• In-process parsing avoids external services at the cost of handling format quirks in code.

🚀Next Steps

• Implement PDF content extraction to match EPUB parsing depth
• Strengthen cross-device sync conflict resolution for the opt-in Postgres path
• Broaden podcast directory support beyond the Apple lookup API