TRD — Local Avatar Render Providers

Status: Draft · Repo: omniavatar-core · Implements: render.Provider

Scope

Technical design for local, offline avatar render providers that implement the existing render.Provider interface. Each provider wraps one complete audio-driven render engine (LatentSync, EchoMimic, …) running as a persistent Python/MLX gRPC server. This document covers the interface mapping, provider granularity, transport, Python workers, on-disk avatar bundle, code placement, and the decision records behind each. It does not restate product rationale (see PRD) or execution sequencing (see PLAN).

Design center

Reuse everything that already exists; add only what is genuinely new.

• The consumer contract already exists: render.Provider + render.AudioUploader. videoascode already codes to it (pkg/avatar/generate.go, cmd/vac/avatar_*, cmd/vac/slides_video.go).
• The transport pattern already exists: gRPC over a Unix Domain Socket to a persistent Python/MLX server, proven by omnivoice-core/providers/{f5tts-mlx,whisper-mlx} — one isolated server + venv per model.
• The registry pattern already exists: the batteries omniavatar module owns RegisterRenderProvider / GetRenderProvider; each providers/<name> registers via init() with thin/thick priority.

The new work is: (1) a shared LocalRender gRPC service, (2) one persistent Python worker per engine, (3) a thin Go adapter per engine mapping render.Provider onto the service, (4) registration in the batteries module, and (5) an on-disk avatar bundle format.

Provider granularity — one provider per complete render engine

The engines do not sit at the same level against the render.Provider contract (audio → MP4):

Engine	Inputs	Complete render provider?	Role
LatentSync	source video + audio	✅ yes	Lip-sync renderer
EchoMimic	portrait + audio	✅ yes (end-to-end)	Alternative renderer
MuseTalk / Wav2Lip	face video + audio	✅ yes	Lip-sync (later options)
LivePortrait	driving video → portrait	❌ no (no audio, no lip-sync)	Motion pre-stage

Consequences:

• Each complete engine is its own provider package and registered name: providers/latentsync → "latentsync", providers/echomimic → "echomimic". This matches the existing cloud providers (named per vendor) and omnivoice-core (named per model). Selection is --avatar-provider latentsync.
• LivePortrait is not a provider. It is an optional motion pre-stage inside a render provider's pipeline (e.g. LatentSync), toggled via Extensions["motion"] = "liveportrait". So LivePortrait + LatentSync compose into one provider; EchoMimic is the end-to-end alternative.
• MVP implements exactly one engine (the spike winner, expected latentsync), structured so a second engine (echomimic) is additive.

Why separate packages + separate servers (not one local provider)

The engines have conflicting Python dependency stacks (LatentSync's diffusion stack vs EchoMimic vs Wav2Lip). Co-locating them in one venv/server invites dependency hell. omnivoice-core already isolates f5tts-mlx and whisper-mlx into separate servers/venvs for exactly this reason; avatar engines follow the same rule. Each provider therefore has its own server/ (venv, requirements, UDS socket).

What is shared

• One proto, proto/localrender/v1 (service LocalRender): the interface is identical across engines (audio → MP4 job), unlike omnivoice's localtts vs localstt which are genuinely different services.
• A shared Go client base (job mapping, local:// upload, bundle resolve), extracted into providers/localrender/ (a library package, not a registered provider) once the second engine lands — not built speculatively for the single-engine MVP.
• The avatar bundle format and the launcher conventions.

Component overview

videoascode (unchanged consumer)
  omniavatar.GetRenderProvider("latentsync") ──► render.Provider
        │  Generate → Wait(Status) → Download   (+ AudioUploader.UploadAudio)
        ▼
omniavatar-core/providers/latentsync   (thin Go gRPC client)
        │  gRPC over unix:///tmp/omniavatar-latentsync.sock
        ▼
latentsync_server.py  (persistent Python/MLX worker, own venv)
   ├── async job queue (Generate returns id; Status polls; Download streams)
   ├── engine: LatentSync  (+ optional LivePortrait motion pre-stage)
   ├── model cache (weights resident across renders)
   ├── avatar bundle loader (idle/source clip + reference)
   └── ffmpeg pre/post (resample audio, build source track, trim, encode)

   ... providers/echomimic + echomimic_server.py: same shape, own socket/venv,
       end-to-end engine, no pre-stage.

Interface mapping (the crux)

The existing render.GenerateRequest is shaped for remote rendering (AvatarID = a provider-hosted avatar, AudioURL = a fetchable URL, or Script = provider TTS). A local renderer has none natively. Rather than invent a new interface (the ideation doc's mistake), map local rendering onto the existing fields. The mapping is identical for every engine provider — only Name() and the socket differ.

render field / method	Local meaning
Provider.Name()	The engine name: "latentsync", "echomimic", …
GenerateRequest.AvatarID	Name of a local avatar bundle (e.g. "john"), resolved to a bundle dir. Required — Validate() already enforces it.
GenerateRequest.AudioURL	A local://… handle returned by our AudioUploader.UploadAudio (below). Exactly one of AudioURL/Script must be set; local uses AudioURL.
GenerateRequest.Script	Unsupported. The avatar stage never does TTS. If set, return ErrInvalidRequest.
GenerateRequest.Width/Height	Output dimensions (default 512×512 for a corner avatar).
GenerateRequest.Background	Best-effort; typically ignored (the videoascode compositor handles circular crop / transparency).
GenerateRequest.Extensions	Per-render knobs: seed (int64), fps (float), reference (bundle asset override), motion (LatentSync only: "liveportrait" to enable the motion pre-stage, or a driving-clip path). No engine key — the engine is the provider identity.
Generate()	Enqueue a job on the engine's worker; return Job{ID, Provider:"latentsync"} immediately (renders are slow; async fits).
Status()	Query the worker; map to JobState (pending/processing/completed/failed); fill Duration, ErrorCode/ErrorMsg, and a local://job/<id>.mp4 VideoURL.
Download()	Stream the finished MP4 from the worker to dst; return ErrJobNotCompleted if not terminal-completed.

Audio: render.AudioUploader as local hosting

videoascode already feature-detects AudioUploader and, when present, calls UploadAudio then passes the returned URL as GenerateRequest.AudioURL (pkg/avatar/generate.go:88, cmd/vac/slides_video.go:256). Each local provider implements AudioUploader as local hosting: UploadAudio copies the stream into a worker-visible scratch dir and returns an opaque local://<sha256>.wav handle; Generate resolves it back to a path. The consumer flow is identical to the cloud providers — no local-vs-remote branch in videoascode.

Optional render.AvatarLister

Implement ListAvatars(ctx, search) to enumerate local avatar bundles (scan the bundles dir, filter by substring), so vac avatar list-avatars --provider latentsync works. Cheap; Phase 2.

Transport & proto

Persistent gRPC server over UDS per engine (unix:///tmp/omniavatar-<engine>.sock), mirroring the voice providers. Persistent because avatar model weights are far heavier than F5-TTS; per-invocation CLI exec would reload gigabytes per segment (this is where the ideation doc's "CLI is fine initially" advice is wrong for our workload).

Shared proto proto/localrender/v1/localrender.proto, service LocalRender:

service LocalRender {
  rpc Generate(GenerateRequest) returns (GenerateResponse);   // returns a job id
  rpc Status(StatusRequest) returns (StatusResponse);         // poll job state
  rpc Download(DownloadRequest) returns (stream VideoChunk);  // stream finished MP4
  rpc ListAvatars(ListAvatarsRequest) returns (ListAvatarsResponse);
  // Health, model lifecycle, runtime info — same shape as localtts/localstt,
  // for operational parity across all local servers.
  rpc Health(HealthRequest) returns (HealthResponse);
  rpc LoadModel(LoadModelRequest) returns (LoadModelResponse);
  rpc UnloadModel(UnloadModelRequest) returns (UnloadModelResponse);
  rpc RuntimeInfo(RuntimeInfoRequest) returns (RuntimeInfoResponse);
}

Every engine server implements the same service; they differ only in the engine behind it and their socket. Generate/Status are unary (async jobs on the server); Download streams (videos are large). Kept deliberately close to proto/localtts and proto/localstt so the local servers share generation scripts, health semantics, and the launcher.

Python workers

providers/<engine>/server/<engine>_server.py, one arm64 venv each, same launch discipline as the voice servers (arch -arm64):

• Async job queue — Generate registers a job and starts a background task; Status reports progress; the finished file is retained for Download.
• Engine — one model per server. LatentSync's server may run an optional LivePortrait motion pre-stage when Extensions["motion"]="liveportrait".
• Model cache — weights load once, stay resident (LoadModel/health).
• Avatar bundle loader — resolves AvatarID → bundle dir; builds a source track of the right duration from the idle clip (loop or concatenate).
• ffmpeg pre/post — resample audio to the engine's rate, build/trim the source video, mux, encode H.264.

On-disk avatar bundle

Referenced by AvatarID; shared across engines. Minimal MVP is a single idle clip + metadata:

~/.omniavatar/avatars/<name>/
    metadata.json        # { "name", "fps"?, "resolution"? }
    idle/
        idle-01.mp4      # neutral, mouth-closed, subtle motion (MVP: one clip)
    references/          # optional; still frames for image-driven engines (EchoMimic)
        front-neutral.png

Bundle root is configurable (env / registry.WithExtension), defaulting to ~/.omniavatar/avatars. Forward-compatible with the richer avatar package the ideation doc sketches (multiple motion clips, references); the MVP loader only needs one idle clip.

Idle source-clip capture guidance (RMI-004)

The lip-sync model overwrites the mouth region and drives it from the audio, so the source clip should be a neutral "listening" take, not a talking one — otherwise residual mouth motion from the source fights the generated lips (the SPIKE-001 render reused a HeyGen talking clip and still worked, but a neutral clip syncs cleaner). Record:

• Duration: one 10–20 s clip for the MVP (loop/concatenate to narration length); build a small library later.
• Mouth: mostly closed, relaxed jaw, occasional slight opening. Do not speak.
• Motion: subtle only — natural blinks, tiny nods/tilts (a few degrees), gentle breathing/shoulder movement. Avoid large head turns (>~15°), looking away, or leaving frame.
• Framing: fixed tripod, constant lighting/exposure, front-facing, clean/plain background, square-ish crop (512²/720²/1080²) for easy circular masking.
• Consistency: same wardrobe/lighting/framing across clips so future cuts and additions match.

Code placement — Decision Records

D1 — Where do the local render providers live? — CONFIRMED

Decision: omniavatar-core/providers/<engine>/ (one package per complete engine), each with its Go gRPC client and its own server/ (venv, requirements, socket), mirroring omnivoice-core/providers/{f5tts-mlx,whisper-mlx}.

• Matches omnivoice-core exactly, making "the local X provider lives in X-core/providers/<model>/" a uniform rule across voice and avatar.
• Separate servers/venvs isolate the engines' conflicting Python dependency stacks — the same reason omnivoice isolates f5tts from whisper.
• Cost: adds google.golang.org/grpc to omniavatar-core's currently-clean go.mod (the same departure omnivoice-core already accepted — grpc v1.82.1). Only packages importing providers/<engine> compile the gRPC code; pure render/live consumers are unaffected at build time, but the module gains the dependency. Action: update docs/architecture.md to read "interfaces + in-tree local providers."
• Alternatives considered: batteries omniavatar/providers/local/ (keeps core dep-free but breaks cross-core symmetry); a new omniavatar-local repo (deferred — no independent consumer/release need yet). Both rejected for now.

D2 — Where does registration happen?

omniavatar-core has no root registry (unlike omnivoice-core); RegisterRenderProvider/GetRenderProvider live in the batteries omniavatar module. Decision: each engine adapter exposes a constructor + registry.RenderProviderFactory in providers/<engine>; batteries omniavatar/providers/<engine>/register.go calls RegisterRenderProvider("<engine>", …) in init() and is blank-imported by omniavatar/providers/all. Same split the SDK render adapters already use, so videoascode (which imports providers/all + calls GetRenderProvider) gets each engine name for free.

D3 — Provider naming & granularity — CONFIRMED

Decision: name providers per model (latentsync, echomimic, later musetalk/wav2lip), not a generic local. Rationale: matches the existing cloud providers (per vendor) and omnivoice (per model); each provider is self-describing; engine is the provider identity, not a hidden Extensions key. LivePortrait is not a provider name — it is a motion pre-stage inside the LatentSync provider (Extensions["motion"]="liveportrait"). Slug for RMIs: OACORE.

Consumer impact (videoascode)

Expected changes: provider selection only. videoascode already gets providers via omniavatar.GetRenderProvider(name, …), feature-detects render.AudioUploader, and runs Generate → render.Wait → Download. To enable local engines: accept latentsync/echomimic as avatar provider values and skip the API-key requirement for them (parallel to the --local flag we added for TTS/STT). The compositor, job flow, and avatar compose path are untouched.

Failure modes & mapping

• Worker unreachable (socket missing) → wrap ErrProviderUnavailable.
• Unknown AvatarID (no bundle) → ErrInvalidRequest.
• Engine/model load failure → job → JobStateFailed with ErrorCode/ErrorMsg; render.Wait surfaces ErrJobFailed.
• Download before completion → ErrJobNotCompleted.
• Script set → ErrInvalidRequest (local render is audio-driven only).

Testing

• Go unit — request mapping (AvatarID/Extensions), local:// upload/resolve, status-state mapping, error wrapping. gRPC client tested against a stub server (skip if no socket, like the whisper-mlx tests).
• Python unit — bundle loader, source-track duration builder, engine contract.
• Manual QA artifact — a checked-in one-slide reference render (as done for the F5-TTS/Whisper demo), reviewed at corner-avatar size.
• Determinism — same audio + seed ⇒ byte-stable (or perceptual-hash-stable) output, for cache/regression.

Risks (technical)

• R1 — Apple Silicon runnability. LatentSync/EchoMimic are CUDA-oriented; MPS support is uncertain. Mitigation: gate all Go work on a runnability spike (RMI-OACORE-001); the shared render.Provider + LocalRender service let us swap engines or point the same provider at a remote NVIDIA worker.
• R2 — Unverified model-spec claims from the ideation doc (VRAM, resolution, fps, repo names — it cites both KlingAIResearch/LivePortrait and the correct KwaiVGI/LivePortrait). Mitigation: verify upstream during the spike.
• R3 — go.mod dependency growth on core. Accepted per D1; contained to the providers/<engine> packages and documented.
• R4 — Weight reload cost. Mitigated by the persistent-server design.
• R5 — Per-engine Python dependency conflicts. Mitigated by one venv/server per engine (the reason for D1's separate-servers rule).

Out of scope

Streaming (live.Provider), TTS, training, full-body. See PRD Non-Goals.