ROADMAP — Local Avatar Render Providers¶
Status: Draft · Repo: omniavatar-core · RMI slug: OACORE
The tracked checklist for the local avatar render providers. Themed phases group
related RMIs; execution and review happen by phase. Phase status is derived
from its member RMIs (never set directly). Commits implementing an item carry the
git trailer Refs: RMI-OACORE-<NNN>.
Status legend: ☐ not started · ◐ in progress · ☑ done · ⊘ blocked
Cross-repo RMIs are tagged: [core] omniavatar-core, [batteries]
omniavatar, [vac] videoascode.
Confirmed decisions (TRD): D1 local providers live in
omniavatar-core/providers/<engine>/ (one package + own server/venv per engine);
D3 providers are named per model (latentsync, echomimic, …), LivePortrait
is a motion pre-stage, not a provider. <engine> below = the MVP engine chosen in
RMI-003 (expected latentsync).
Phase 1 — Engine spike & selection (status: ◐ in progress)¶
The gate. No downstream Go work begins until an Apple-Silicon-runnable engine is
chosen. Throwaway Python only. Findings: spike/SPIKE-001-findings.md.
| RMI | Item | Status |
|---|---|---|
| RMI-OACORE-001 | Spike: run LatentSync, MuseTalk, and Wav2Lip on Apple Silicon (MPS/CPU) against a real narration WAV + idle clip; record actual resource use and latency [core] — ✅ Wav2Lip verified on MPS (~4 min/13.6 s @ 360p), acceptable at corner size | ☑ |
| RMI-OACORE-002 | Verify upstream repos and their real Apple-Silicon status; produce a corrected fact sheet superseding IDEATION_CHAT_LOCAL.md (repo names, VRAM, resolution, fps) [core] |
☑ |
| RMI-OACORE-022 | MuseTalk MPS spike — verify the commercial-licensed (MIT) engine runs on Apple Silicon [core] — ☑ negative: blocked by mmcv/DWPose (no arm64 wheel, source build fails) + unresolved macOS support. See spike/SPIKE-022-musetalk.md |
☑ |
| RMI-OACORE-023 | Research commercial + MPS engine (avoid openmmlab/CUDA) [core] — ☑ SadTalker is the best on-device candidate (Apache code, MPS-fast via PIRender, no mmcv) but bundles non-commercial components → needs license diligence. See spike/SPIKE-023-alt-engine-research.md |
☑ |
| RMI-OACORE-024 | SadTalker MPS spike [core] — ☑ runs on MPS (PIRender ~8 fps, 256² talking head) with no openmmlab; but pipeline uses non-commercial Deep3DFaceRecon (load-bearing) + Wav2Lip → not commercial-clean as-is. See spike/SPIKE-023 |
☑ |
| RMI-OACORE-025 | Extended engine research (commercial × Apple-Silicon) [core] — ☑ found the MLX-native path: LTX-2 is tiered-commercial (free < $10M ARR), MLX-native, no BFM/InsightFace/openmmlab; JoyVASA+LivePortrait (MIT) is a lower-cost fork (swap InsightFace). See engine-landscape | ☑ |
| RMI-OACORE-027 | LivePortrait + JoyVASA spike (the fully-free path) [core] — ☑ SUCCESS: MIT, audio-driven 512² talking head runs on MPS (SPIKE-027); commercial gate = swap InsightFace→mediapipe. The fully-free on-device winner. | ☑ |
| RMI-OACORE-026 | LTX-2 MLX spike — deprioritized: LTX-2's license is revenue-capped (not fully-free), which the team ruled out [core] | ☐ |
| RMI-OACORE-003 | Choose the first engine to build [core] — ☑ decided: providers/liveportrait-joyvasa (fully-free MIT, MPS-proven in SPIKE-027). Commercial path: InsightFace→mediapipe swap. Port patches: cuda→mps, torch 2.3.0, curated deps. |
☑ |
| RMI-OACORE-004 | Define the avatar bundle format (metadata.json + idle clip) and build a reference john bundle; idle-clip capture guidance recorded in TRD [core] — ☑ avatar/bundle.go with Load/List/Metadata types; avatars/example/ reference bundle; capture guidance in TRD |
☑ |
Phase 2 — Shared proto & the first engine's Python worker (status: ☐ not started)¶
Persistent gRPC/UDS server for the chosen engine, with its own venv/socket, mirroring the localtts/localstt providers.
| RMI | Item | Status |
|---|---|---|
| RMI-OACORE-005 | Author the shared proto/localrender/v1/localrender.proto (LocalRender: Generate/Status/Download/ListAvatars/Health/LoadModel/UnloadModel/RuntimeInfo) [core] — ☑ proto authored with full service/message definitions |
☑ |
| RMI-OACORE-006 | Scaffold providers/<engine>/server/ (own venv + requirements); generate stubs; extend the launcher to start it under arch -arm64 on its own socket [core] — ☑ providers/liveportrait-joyvasa/server/ with requirements.txt, run.sh, generate_proto.sh |
☑ |
| RMI-OACORE-007 | Implement <engine>_server.py: async job queue + Health/LoadModel/RuntimeInfo [core] — ☑ joyvasa_server.py with LocalRender gRPC service, job queue, lifecycle methods |
☑ |
| RMI-OACORE-008 | Wire the chosen engine + model cache (weights resident across renders) [core] — ☑ inference.py with JoyVASAPipeline (HuBERT, DiT, LivePortrait, mediapipe); model caching across renders |
☑ |
| RMI-OACORE-009 | Implement avatar-bundle loader + ffmpeg source-track/duration builder + H.264 encode [core] — ☑ integrated in inference.py: source frame extraction, ffmpeg muxing with audio |
☑ |
Phase 3 — Go adapter (render.Provider) (status: ☑ done)¶
Thin gRPC client in providers/<engine>, mapping onto the existing interfaces.
| RMI | Item | Status |
|---|---|---|
| RMI-OACORE-010 | providers/<engine>: gRPC client scaffolding, Name() = engine name, connection lifecycle, Close() [core] — ☑ provider.go with grpc.NewClient, Name(), Close() |
☑ |
| RMI-OACORE-011 | Map Generate / Status / Download onto render.Provider (job-state mapping, error wrapping to render sentinels) [core] — ☑ mapJobState(), wrapGRPCError() with sentinel mapping |
☑ |
| RMI-OACORE-012 | Implement render.AudioUploader (local local:// hosting) + Extensions knobs (seed/fps/reference/motion) [core] — ☑ UploadAudio() with local:// scheme, Extensions for seed/fps/motion_scale |
☑ |
| RMI-OACORE-013 | Optional render.AvatarLister; compile-time interface assertions; unit tests against a stub server [core] — ☑ ListAvatars(), compile-time var _ checks |
☑ |
Phase 4 — Registration & consumer wiring (status: ☑ done)¶
Make it selectable end-to-end. Deliberately the smallest phase — the payoff of conforming to existing interfaces.
| RMI | Item | Status |
|---|---|---|
| RMI-OACORE-014 | Expose constructor + registry.RenderProviderFactory for <engine>; update docs/architecture.md for the in-tree local providers (Decision D1) [core] — ☑ NewFromConfig() factory function |
☑ |
| RMI-OACORE-015 | Add omniavatar/providers/<engine>/register.go (RegisterRenderProvider("<engine>", …)) and include it in providers/all [batteries] — ☑ omniavatar v0.3.0 |
☑ |
| RMI-OACORE-016 | Allow <engine> as an avatar provider in videoascode; skip the API-key requirement; add selection flags [vac] — ☑ videoascode: --provider liveportrait-joyvasa with localProviders map |
☑ |
| RMI-OACORE-017 | End-to-end offline demo (F5-TTS → local avatar → Whisper → composite); check in a one-slide reference render [vac] — ☑ examples/local-avatar/ with presentation.md and README |
☑ |
Phase 5 — Hardening, second engine & docs (status: ☐ not started)¶
| RMI | Item | Status |
|---|---|---|
| RMI-OACORE-018 | Determinism (seed) + render caching; regression artifact [core] | ☐ |
| RMI-OACORE-019 | Extract the shared Go client base (providers/localrender/) and add the second engine providers/echomimic (own server/venv) to prove the structure is additive [core] |
☐ |
| RMI-OACORE-020 | LivePortrait motion pre-stage inside the LatentSync server (motion=liveportrait); launcher polish (status/stop), .gitignore for venvs/stubs/models [core] |
☐ |
| RMI-OACORE-021 | Docs: per-provider READMEs, docs/local-render guide, mkdocs nav; fold in the corrected fact sheet from RMI-002; update CHANGELOG [core] |
☐ |
Notes¶
- Gate: Phase 1 must complete before Phase 2+. If no engine clears the
Apple-Silicon bar, RMI-003 records the decision to run inference on a remote
NVIDIA worker behind the same
render.Provider+LocalRenderservice; Phases 2–5 proceed with the server pointed at that worker (see PLAN "Contingency"). - Confirmed decisions: D1 (placement) and D3 (per-model naming) are settled;
RMI-014 carries out the
architecture.mdupdate they imply. D2 (registration in batteries) is defined in TRD. - MVP = one engine. Phases 2–4 build a single engine provider end-to-end; Phase 5 proves additivity with a second engine and the shared base.