Skip to content

PRD: OmniAvatar Render — Batch Avatar Video Generation

Status: Approved Date: 2026-07-16 Owner: PlexusOne Related: TRD.md, PLAN.md, ROADMAP.md. Origin: internal ideation notes (IDEATION_CHAT_PRESENTATION.md, untracked)

Overview

OmniAvatar today supports one product surface: real-time streaming avatars (LiveKit sessions with PCM audio streaming for lip-sync), used by OmniMeet-style conversational agents. This PRD adds a second surface: batch (asynchronous) avatar video generation — submit narration audio, receive a talking-head MP4.

The primary consumer is videoascode (vac), which converts Marp presentations with voiceovers into videos. The batch render capability lets vac overlay a circular talking-head presenter (driven by the exact OmniVoice narration audio) onto slide videos.

Problem Statement

Presentation videos generated by videoascode have narration audio but no visible presenter. Adding a talking-head overlay increases engagement and perceived production quality (YouTube, Udemy, internal enablement). Today there is no provider-agnostic Go interface for "audio in → talking-head video out"; each provider has a different API shape, avatar identity model, and audio-delivery mechanism.

Goals

  1. Provide a provider-agnostic Go interface for asynchronous avatar video generation: submit → poll → download.
  2. Drive avatar lip-sync from existing narration audio (the same audio used in the final presentation), not provider TTS, so pauses, pronunciation, and slide timing match exactly.
  3. Implement the interface for three providers — HeyGen, Tavus, bitHuman — to validate the abstraction ("rule of three") before freezing it.
  4. Keep omniavatar-core dependency-free (interfaces only); provider implementations live in omniavatar.
  5. Rename the existing real-time package so the two surfaces are self-describing: live/ (real-time sessions) and render/ (batch generation).

Non-Goals

  • Real-time/streaming avatar changes (the live/ surface is renamed but functionally unchanged).
  • Video composition (circle masking, overlay onto slides) — this stays in videoascode via FFmpeg, keeping the render output provider-neutral.
  • Provider-side TTS as the primary path (text scripts are supported as a secondary input, but audio-driven generation is the design center).
  • Webhook/callback-based job completion (polling first; webhooks are on the roadmap).
  • D-ID or other additional providers (roadmap).

Use Cases

UC1: Presentation presenter overlay (primary)

vac concatenates per-slide narration into one audio file, calls render.Provider.Generate with the audio URL and an avatar ID, polls until complete, downloads presenter.mp4, then composites a circular crop onto slides.mp4 locally with FFmpeg.

UC2: Standalone talking-head clips

Generate short narrated clips (announcements, greetings, course intros) from audio or a text script without any slide pipeline.

UC3: Provider portability

Switch providers (cost, quality, licensing) by changing one registry name and the avatar identity extension — no pipeline changes.

Provider Landscape

All three providers already have Go SDKs in the PlexusOne org, and all three support audio-driven asynchronous video generation:

Capability HeyGen (heygen-go) Tavus (tavus-go) bitHuman (bithuman-go)
Async video generation video.Generate (v2) CreateVideo CreateVideo
Drive with existing audio (URL) Yes Yes (.wav/.mp3) Yes
Text-script alternative Yes (voice_id) Yes (script) Yes (voice_id)
Poll status → MP4 URL Yes Yes Yes
Upload local audio file Yes (asset package, added with this work) No (no upload API) Yes (UploadFile)
Avatar identity avatar_id / talking_photo_id replica_id agent_id

The uneven upload support is a first-class design input: the core interface treats audio-by-URL as the common denominator and models upload as an optional capability. See TRD.md.

Requirements

Functional

  • FR1: Generate(ctx, req) submits a job and returns a provider job ID without blocking on completion.
  • FR2: Status(ctx, jobID) returns a normalized job state (pending, processing, completed, failed) plus the raw provider status string.
  • FR3: Download(ctx, jobID, dst) streams the completed MP4 to an io.Writer.
  • FR4: Requests accept AudioURL (primary) or Script text (secondary); providers that need a voice for scripts read it from extensions.
  • FR5: Providers that can host audio implement an optional AudioUploader capability; callers can feature-detect it. Providers that cannot return a typed sentinel error.
  • FR6: Providers register in a global registry symmetric with the live registry (GetRenderProvider("heygen", opts...)), with thin/thick priority semantics.
  • FR7: A polling helper waits for terminal state with configurable interval, honoring context cancellation.

Non-Functional

  • NFR1: omniavatar-core gains no new third-party dependencies (stdlib only).
  • NFR2: Breaking changes are acceptable (v0.x) but must be documented in release notes for both repos.
  • NFR3: Job IDs, states, and errors must be loggable/serializable for caching by consumers (e.g., vac caches by audio-hash + avatar config).
  • NFR4: All errors follow the house error-handling rules (typed sentinels, wrapped provider errors; never silently discarded).

Success Criteria

  1. omniavatar-core exposes live/ and render/ packages; both compile with no provider dependencies.
  2. omniavatar registers three render providers; GetRenderProvider returns a working provider for each name.
  3. bitHuman provider demonstrates the AudioUploader capability end-to-end (base64 upload → hosted URL → generation).
  4. Tavus provider demonstrates the URL-only path with ErrAudioUploadUnsupported behavior.
  5. videoascode can implement UC1 against the interface without provider-specific imports (integration itself is on the roadmap).

Naming Decision

The existing omniavatar-core/avatar package is renamed to omniavatar-core/live. Rationale: with two surfaces, avatar.Provider vs render.Provider does not communicate which mode is which — "avatar" is the umbrella concept (the module name), not a mode. live.Session / render.Job are self-describing at every call site and mirror vendor vocabulary (HeyGen "LiveAvatar" vs. "Video Generation"). Alternatives considered: realtime/ (longer at every call site), stream/ (overloaded with audio/HTTP streams).