Skip to content

PLAN — Local Avatar Render Provider

Status: Draft · Repo: omniavatar-core · Companion: ROADMAP.md (the checklist)

This is the execution narrative: the approach, why the work is ordered the way it is, what the critical path and dependencies are, and how we validate and roll out. The tracked, per-item checklist lives in ROADMAP.md; this document explains the why behind that sequence and does not restate it item-by-item.

Guiding approach

Build inward-out from what already works. The consumer contract (render.Provider), the transport pattern (gRPC/UDS + persistent Python/MLX server), and the registry are all proven — by the cloud avatar providers and by the local F5-TTS/Whisper voice providers. The genuinely new and risky part is narrow: does any lip-sync engine run acceptably on Apple Silicon? So we de-risk that first, in Python, before writing a line of Go adapter code, and we keep the engine behind an adapter so the answer can change without disturbing the interface.

Two principles shape the whole plan:

  1. Prove the engine before plumbing it. No Go work, no proto, no interface mapping matters if no engine runs on the target hardware. The spike is the gate.
  2. Conform, don't invent. Every Go-facing decision maps onto the existing render interfaces (see TRD). Success is measured by videoascode needing only provider-selection changes.

Critical path

Engine spike (MPS)  ─►  proto + Python worker  ─►  Go adapter  ─►  registration
   (gate)                 (persistent server)      (render.Provider)   (batteries)
                                                          │
                                                          ▼
                                              videoascode selection + demo

The engine spike is the sole gate. Everything downstream assumes a chosen, runnable engine. If the spike shows nothing runs acceptably on Apple Silicon, the plan branches (see "Contingency") to a remote NVIDIA worker — but crucially, behind the same interface, so the downstream work is largely unchanged.

Why this order

  • Spike first (Phase 1). The largest uncertainty by far is hardware runnability and output quality at corner size, not interface design. We answer it with throwaway Python: run LatentSync, MuseTalk, and Wav2Lip on a real narration WAV + idle clip, on MPS/CPU, and judge quality composited at 180–320 px. Output: the one engine we build first (each engine is its own provider package + server; MVP ships one, expected latentsync) and its real resource/latency numbers (replacing the ideation doc's unverified figures). Verify the upstream repos and their actual Apple-Silicon status here, too.
  • Transport + worker before Go (Phase 2). Once an engine is chosen, stand up the persistent gRPC server around it, reusing the localtts/localstt proto and launcher conventions. Proving the async job lifecycle (Generate→Status→Download) in Python — with the model resident — is what makes the Go adapter trivial.
  • Go adapter after the wire is real (Phase 3). With a working service, the Go side is a thin, well-understood mapping onto render.Provider + AudioUploader. Doing it after Phase 2 means we implement against a live server, not a guess.
  • Registration + consumer wiring last (Phase 4). Only once the adapter is real do we register the engine (e.g. "latentsync") in batteries and enable it in videoascode. This is deliberately the smallest, last step — the proof that "conform, don't invent" paid off is that this step is nearly trivial.
  • Polish and hardening (Phase 5). Determinism, caching, AvatarLister, the launcher script, and docs — after the happy path works end-to-end.

Dependencies

  • Upstream models (LatentSync / MuseTalk / Wav2Lip) — external; their Apple-Silicon behavior is the spike's subject.
  • omnivoice-core local-provider patterns — reused as the template for proto, server, and launcher. No code dependency, but a strong convention dependency: keep the three local services (tts/stt/render) operationally parallel.
  • Batteries omniavatar module — where registration lands (D2). A cross-repo change coordinated with the core adapter.
  • videoascode — the consumer; final wiring + demo. Also cross-repo.
  • ffmpeg — already a videoascode dependency; reused for pre/post.

The cross-repo touch points (core adapter → batteries registration → videoascode selection) are small and sequential; they are called out in the ROADMAP as distinct RMIs so the coordination is explicit.

Validation strategy

  • Spike acceptance — a human judges the reference render at corner size. The bar is "acceptable presenter at 180–320 px," not full-frame fidelity. This is a judgment call, deliberately made by a person, and recorded with the chosen engine's numbers.
  • Interface conformance — the Go adapter satisfies render.Provider and render.AudioUploader (compile-time interface assertions, as the voice providers do).
  • End-to-end proof — the definition of done is SC1/SC2 from the PRD: the fully local presentation path (F5-TTS → local avatar → Whisper → composite) runs offline, and a one-slide reference MP4 is checked in as a manual QA artifact (exactly as we did for the F5-TTS/Whisper demo).
  • Determinism — same inputs + seed produce stable output, so caching and regression comparisons are meaningful.

Rollout

Local rendering is opt-in and additive; it changes nothing for existing cloud users. It ships behind explicit per-model provider selection (e.g. --avatar-provider latentsync), alongside the cloud provider names. The persistent servers are developer-launched (the launcher script), never started implicitly. Because the provider is registered only when providers/all (or an explicit local import) is compiled in, and selected only by name, there is no risk to the default cloud path.

Risks & mitigations (execution)

  • The engine may not run acceptably on Apple Silicon. This is the headline risk and the reason for the gate. Mitigation: the spike decides before we invest in Go; the adapter keeps the engine swappable.
  • Contingency — remote NVIDIA worker. If no local engine clears the bar, the same render.Provider adapter points at a gRPC worker on an NVIDIA box instead of a local socket. The Go interface, videoascode wiring, and job lifecycle are unchanged; only the server's location and the transport endpoint differ. This is why we invest in the interface mapping regardless of where inference runs.
  • Ideation-doc numbers are unverified. We do not carry any VRAM/FPS/resolution figure or repo name from IDEATION_CHAT_LOCAL.md into implementation without confirming it during the spike (the doc even cites a wrong LivePortrait org).
  • Cross-repo coordination drift. The core adapter, batteries registration, and videoascode selection are three repos. Mitigation: they are ordered and small, and each is a named RMI so none is silently skipped.
  • Scope creep toward LivePortrait / motion generation. Tempting but not needed for a corner presenter. LivePortrait is a motion pre-stage inside the LatentSync provider (motion=liveportrait), not its own provider; it is held to Phase 5+ / future work. The MVP is one lip-sync engine over a looped idle clip. Adding a second engine (echomimic) is additive, not a rewrite.

Definition of done (this effort)

The local avatar provider is "done" for v1 when: an Apple-Silicon-runnable engine is chosen and documented; omniavatar.GetRenderProvider("<engine>") (e.g. "latentsync") returns a working render.Provider; videoascode produces a composited presenter from local narration with provider-selection-only changes; the fully-offline presentation path works; and a reference render is checked in. Everything beyond that (LivePortrait motion, richer avatar bundles, remote-worker productionization) is explicitly future work.