Local Avatar Engine Landscape¶
A synthesis of which lip-sync / talking-head engines can back the local avatar
render provider, evaluated against the two constraints that actually bind our
choice: commercial licensing (intent is commercial) and local execution on
Apple Silicon (M1 Max dev machine, MPS). Detailed spike logs live under
spike/.
The binding constraint¶
The engine must be both commercially licensable and runnable locally on Apple Silicon. Two independent walls knock out the obvious candidates:
- openmmlab (mmcv / mmpose / mmdet): used by DWPose-based face preprocessing. No arm64 macOS wheel; source build fails. Blocks MuseTalk.
- CUDA-only: diffusion stacks written for NVIDIA with no MPS path. Blocks LatentSync.
Engine matrix¶
| Engine | License / commercial | Local on Apple Silicon (MPS) | openmmlab? | Status |
|---|---|---|---|---|
| Wav2Lip | ❌ non-commercial (LRS2 weights) | ✅ proven (SPIKE-001, ~4 min/13.6 s @ 360p) | no | Dev/reference only — not shippable |
| MuseTalk | ✅ MIT (code + weights) | ❌ mmcv/DWPose won't build on arm64 (SPIKE-022) | yes | Blocked locally; viable only via remote worker or a DWPose-replacement fork |
| LatentSync | ✅ Apache-2.0 | ❌ CUDA-only (8–18 GB VRAM) | no | Clean commercial — remote NVIDIA worker path |
| SadTalker | ⚠️ Apache-2.0 code; bundles non-commercial components | ✅ runs via PIRender facerender (~8 fps @ 256², SPIKE-023) | no (face_alignment) |
Runs on MPS, but 3DMM = non-commercial Deep3DFaceRecon (load-bearing) |
| LTX-2 (Lightricks) | ✅ tiered-commercial — free < $10M ARR, paid above (OpenRAIL open-weights) | ✅ MLX-native (ltx-2-mlx) — cleanest Apple-Silicon path |
no | Most promising new lead — no BFM/InsightFace/Wav2Lip; cost = heavy RAM (~50 GB) + slow offline render |
| LivePortrait + JoyVASA | ⚠️ MIT code; uses non-commercial InsightFace models | ✅ PROVEN on MPS (SPIKE-027: 512² audio-driven talking head) | no (implicit keypoints, no BFM) | The fully-free path. Swap InsightFace→mediapipe for commercial (LivePortrait's LICENSE documents it); MIT/Apache end-to-end after |
| Hallo2/3, SyncTalk, EchoMimic, InfiniteTalk | mixed | ❌ CUDA / NeRF-heavy | some | Not Apple-Silicon MVPs |
| TalkingHead (met4citizen) | ✅ MIT | ✅ (JS/WebGL) | no | 3D avatar, not a photoreal real-person face — wrong category |
The core finding¶
Among the PyTorch/MPS lip-sync engines, none is both cleanly commercial and local-on-Apple-Silicon: they split into MPS-friendly-but-non-commercial (Wav2Lip; SadTalker's bundled Deep3DFaceRecon; JoyVASA's InsightFace) versus cleanly-commercial-but-CUDA/openmmlab-bound (LatentSync, MuseTalk). The recurring blocker is a single non-commercial pretrained component — LRS2 weights (Wav2Lip), Basel Face Model (SadTalker), or InsightFace models (JoyVASA) — or an arm64-hostile one (openmmlab DWPose, MuseTalk).
The exception is the MLX-native diffusion path. LTX-2 (Lightricks) is tiered-commercial (free under $10M ARR), runs natively on Apple Silicon via MLX, and carries none of those non-commercial components. It shifts the problem from "licensing/openmmlab blocker" to "compute weight" (RAM + render time) — a far more tractable tradeoff for offline presentation rendering, and strategically aligned with the existing MLX F5-TTS/Whisper stack.
Is there a fully-free engine on Mac ARM64?¶
"Commercially usable" and "fully free" are different bars. Three license tiers appear here:
- Fully free — permissive (MIT/Apache/BSD), no revenue cap, no non-commercial sub-component.
- Tiered-commercial — free below a revenue threshold, paid above (LTX-2's OpenRAIL: free < $10M ARR).
- Non-commercial — research/personal only (Wav2Lip's LRS2 weights; SadTalker's Deep3DFaceRecon/BFM; JoyVASA's InsightFace models).
Off the shelf, no engine is simultaneously fully-free, Apple-Silicon-native, and free of non-commercial sub-components. The fully-free-licensed engines each hit one obstacle on ARM64:
| Engine | Fully-free license? | Obstacle on Apple Silicon |
|---|---|---|
| MuseTalk | ✅ MIT code + commercial weights | openmmlab DWPose won't build on arm64 |
| LatentSync | ✅ Apache-2.0 | CUDA-only — no MPS |
| LivePortrait | ✅ MIT | runs on MPS, but InsightFace face models are non-commercial + it needs an audio driver |
| LTX-2 | ⚠️ tiered ( $10M ARR cap) | none technical — MLX-native |
So the fully-free + on-Apple-Silicon answer requires a one-component swap, not a new model: replace MuseTalk's DWPose, or swap InsightFace in LivePortrait/JoyVASA, with a permissively-licensed detector. That single fork yields an engine that is MIT/Apache end-to-end and runs on the M1. LTX-2 avoids the fork but trades it for a revenue-cap license.
This is now proven, not theoretical. SPIKE-027 ran LivePortrait + JoyVASA
end-to-end on the M1 — a 512² audio-driven talking head from a single frame + our
F5-TTS narration, both MIT — with only mechanical port patches (torch 2.3 for
Conv3D fallback, cuda→mps device literals, curated deps). The remaining step to
fully-free commercial is the documented InsightFace→mediapipe swap. So the
fully-free on-device path is a maintainable fork of a working pipeline, not a
research bet.
SadTalker — the candidate under evaluation¶
The closest to threading both constraints:
- Runs on MPS with the PIRender lightweight facerender (
--facerender pirender) — the defaultfacevid2viduses Conv3d, which MPS doesn't support (~50 min/3 s on CPU); PIRender replaces it (~5 s/3 s on MPS, ~100× faster). 256×256 only, minor quality loss — acceptable at corner-avatar size. - No openmmlab — uses
face_alignment,facexlib,GFPGAN. - Licensing asterisk — the code is Apache-2.0, but the inference path bundles Deep3DFaceReconstruction (3DMM coefficients; likely load-bearing) and Wav2Lip (lip enhancer; likely optional), both research/non-commercial. This is the mirror image of MuseTalk: instead of a mmcv technical blocker, a licensing one — and the resolution is the same shape, component replacement.
Component-replacement strategy¶
Both blocked commercial engines can, in principle, be unblocked by swapping the offending component:
- MuseTalk: replace openmmlab DWPose with a mac-friendly landmark detector
(
face_alignment/ mediapipe / insightface) → unblocks MPS. - SadTalker: determine whether the non-commercial components are load-bearing, and replace/remove them (drop the Wav2Lip enhancer; replace or clean-license the Deep3DFaceRecon 3DMM stage) → unblocks commercial use.
Whether either is worth the fork depends on how load-bearing the component is and the ongoing maintenance cost. The SadTalker spike (RMI-003 work) measures exactly which components load and how removable they are.
Paths forward (all behind the same render.Provider)¶
- LTX-2 via MLX (recommended next spike) — on-device and commercially licensed with no non-commercial components; MLX-native like F5/Whisper. Unknown to verify: whether it renders acceptably (RAM + speed) on the M1 Max 64 GB and whether corner-avatar quality justifies the compute. Best chance at a clean on-device + commercial MVP.
- JoyVASA + LivePortrait (MIT), with InsightFace swapped — on-device + commercial after replacing the non-commercial InsightFace detector with a permissive one (lower fork cost than SadTalker's 3DMM). MPS unproven for JoyVASA specifically.
- Remote NVIDIA worker, LatentSync (Apache) — cleanly commercial, high quality, no fork, but not on-device. The reliable fallback.
- SadTalker + PIRender — runs on MPS today but requires clearing/replacing the non-commercial Deep3DFaceRecon 3DMM stage; higher fork cost than JoyVASA.
The provider architecture (proto, server, Go adapter, bundle) is identical across all engines and both runtimes; only the engine module and where it runs differ. An MLX-native engine (LTX-2) would additionally reuse the arm64/MLX server discipline already built for F5-TTS and Whisper.