Skip to content

PRD — Local Avatar Render Provider

Status: Draft · Owner: PlexusOne · Repo: omniavatar-core

Summary

Add a local, offline avatar render provider to the OmniAvatar ecosystem so that a talking-head presenter video can be produced entirely on-device, with no cloud API, no per-minute cost, and no narration audio leaving the machine. The provider implements the existing render.Provider interface so that videoascode and any other OmniAvatar consumer can select it exactly like the HeyGen, Tavus, and bitHuman render providers — by name, with no pipeline changes.

This mirrors what we already shipped for speech: local F5-TTS and Whisper providers behind the same OmniVoice interfaces used for ElevenLabs and Deepgram. Local avatar rendering is the avatar-stage counterpart.

Background

Today videoascode composites a small corner presenter into slide videos using OmniAvatar's render surface (render.Provider: Generate → Status/Wait → Download), backed by cloud providers (HeyGen/Tavus/bitHuman). The audio comes from OmniVoice (now optionally local F5-TTS); the avatar stage still requires a cloud account and uploads the narration audio off-device.

An ideation transcript (IDEATION_CHAT_LOCAL.md, repo root) explored a local replacement. That transcript predates and is unaware of the current omniavatar-core interfaces; this PRD supersedes it and is scoped to extend the existing code, not introduce parallel abstractions.

Goals

  • G1 — Produce a lip-synced talking-head MP4 locally from (a) narration audio and (b) a reusable avatar source, suitable for a 180–320 px corner presenter.
  • G2 — Expose it through the existing render.Provider interface and the omniavatar registry, selectable by model name (e.g. --avatar-provider latentsync) alongside the cloud providers, with no changes to the videoascode compositor.
  • G3 — Run on the primary dev machine: Apple Silicon (M1 Max, 64 GB), reusing the established Go-orchestration → Python/MLX-inference pattern.
  • G4 — Keep narration audio and identity assets on-device (privacy) and incur zero per-render cost.
  • G5 — Deterministic, cacheable, per-segment rendering that slots into the existing per-slide artifact model.

Non-Goals

  • NG1 — Real-time / streaming avatars (the live.Provider surface). Out of scope; this PRD is render-only.
  • NG2 — In-house model training or avatar fine-tuning. We consume existing open models.
  • NG3 — Full digital humans (hands, body, scene). Corner head-and-shoulders only.
  • NG4 — Replacing OmniVoice TTS. The avatar stage consumes finished narration audio; it never does its own text-to-speech.
  • NG5 — Guaranteeing production-grade real-time throughput on Apple Silicon. Offline render latency is acceptable; a remote NVIDIA worker is an allowed fallback behind the same interface.

Users & use cases

  • Presentation authors running videoascode who want a presenter avatar without a HeyGen/Tavus subscription or who cannot send audio to a third party.
  • PlexusOne developers iterating locally who want the avatar stage to work offline alongside local F5-TTS/Whisper.

Primary use case:

Given narration WAV (from OmniVoice) and a reusable local avatar, produce a lip-synced corner-presenter MP4 that videoascode composites into a slide presentation — identical output shape to today's HeyGen path.

Requirements

Functional

  • FR1 — Accept narration audio (WAV/MP3) and a reference to a local avatar identity, and produce a talking-head MP4.
  • FR2 — Implement render.Provider (Name, Generate, Status, Download) so consumers use the existing Generate → Wait → Download flow.
  • FR3 — Accept local audio without a public URL by implementing the optional render.AudioUploader capability (local hosting), so videoascode's existing upload-then-generate flow works unchanged.
  • FR4 — Be discoverable through the omniavatar registry per model (GetRenderProvider("latentsync", …), GetRenderProvider("echomimic", …)).
  • FR5 — Support a small avatar asset bundle (a reusable idle/source clip plus metadata) referenced by name, so one identity is reused across renders.
  • FR6 — Allow engine and render options (engine choice, seed, resolution, fps, reference/motion overrides) without breaking the provider-agnostic request — carried in GenerateRequest.Extensions.
  • FR7 — Report progress and terminal state through Status (pending / processing / completed / failed) with a usable error message on failure.

Non-functional

  • NFR1 — First render on Apple Silicon completes within a few minutes for a ~15–60 s segment (offline expectation, not real-time).
  • NFR2 — Output quality is acceptable at corner-avatar display size (evaluated composited at 180–320 px, not full-frame).
  • NFR3 — Model weights load once and stay resident across renders (no per-segment multi-GB reload).
  • NFR4 — Deterministic given the same inputs + seed (for caching and regression tests).
  • NFR5 — Go-side footprint stays thin (a gRPC client); all ML complexity is isolated in the Python worker, matching the F5/Whisper providers.

Success criteria

  • SC1 — vac avatar generate --provider latentsync --avatar-id <name> --audio narration.wav produces a presenter MP4 that vac avatar compose overlays onto a slide video, with zero videoascode code changes beyond provider selection.
  • SC2 — The full local presentation path works offline end-to-end: F5-TTS (audio) → local avatar (presenter) → Whisper (subtitles) → composite.
  • SC3 — A one-slide reference render is checked in as a manual QA artifact, as we did for the F5-TTS/Whisper demo.
  • SC4 — At least one lip-sync engine is confirmed to run on Apple Silicon (MPS/CPU) at acceptable corner-avatar quality, or a documented remote-NVIDIA fallback is in place.

Open questions

  • Q1 — Which engine is the MVP renderer on Apple Silicon (LatentSync vs MuseTalk vs Wav2Lip vs EchoMimic)? Resolved by the MPS-runnability spike; see TRD/ROADMAP.
  • Q2 — Where do avatar asset bundles live on disk, and what is the minimal metadata.json? See TRD.

Resolved: provider granularity and naming — one provider per complete engine, named per model (latentsync, echomimic); LivePortrait is a motion pre-stage, not a provider (TRD Decisions D1/D3).

References

  • Existing interfaces: render/ package in this repo (Provider, AudioUploader, AvatarLister, GenerateRequest, Job/JobStatus/Wait).
  • Precedent: omnivoice-core/providers/{f5tts-mlx,whisper-mlx} (local gRPC-over-UDS providers + Python/MLX servers).
  • Consumer: videoascode pkg/avatar, pkg/orchestrator, cmd/vac/avatar_*.
  • Superseded ideation: IDEATION_CHAT_LOCAL.md.