Skip to content

SPIKE-023 — Alternative commercial + MPS engine research

RMI: RMI-OACORE-023 · Status: ☑ research done · Trigger: "research alt MPS engine first"

Goal: find a lip-sync / talking-head engine that is commercially licensed AND runs locally on Apple Silicon without the openmmlab (mmcv/DWPose) or CUDA-only blockers that ruled out MuseTalk and LatentSync.

Engine landscape (commercial × local-MPS)

Engine License / commercial Local on M1 (MPS) openmmlab? Verdict
Wav2Lip ❌ non-commercial (LRS2) ✅ proven no dev/reference only
MuseTalk ✅ MIT ❌ mmcv/DWPose won't build yes blocked (SPIKE-022)
LatentSync ✅ Apache-2.0 ❌ CUDA-only no remote-NVIDIA only
SadTalker ⚠️ Apache-2.0 code, but bundles non-commercial components ✅ fast on MPS via PIRender no (uses face_alignment) best on-device candidate — needs license diligence
Hallo2/3, SyncTalk, EchoMimic mixed ❌ CUDA/NeRF-heavy some not M1
TalkingHead (met4citizen) ✅ MIT ✅ (JS/WebGL) no 3D RPM-style avatar, not a photoreal real-person face — wrong category

SadTalker specifics (the candidate)

  • License: code Apache-2.0, non-commercial restriction removed. Asterisk: README says it "uses models from Deep3DFaceReconstruction and Wav2Lip; review those licenses separately." Both are research/non-commercial. So as shipped, SadTalker's inference path includes non-commercially-licensed third-party models. Commercial use hinges on whether those components are load-bearing or removable — diligence required (Deep3DFaceRecon is used for 3DMM coefficient extraction, likely load-bearing; Wav2Lip is an optional lip enhancer, likely removable).
  • MPS: the default facevid2vid renderer uses Conv3d, which MPS doesn't support (~50 min/3 s on CPU). The PIRender lightweight facerender (--facerender pirender, PR #458) replaces it and runs ~5 s / 3 s audio on MPS (~100× faster). Limits: 256×256 only, minor quality loss (teeth). 256² is fine for a corner avatar.
  • Deps: facexlib, GFPGAN, face_alignment — no openmmlab/mmcv. Avoids the MuseTalk wall.
  • Shape: single image + audio → generated talking head (end-to-end, EchoMimic-style), not lip-sync-over-video. Fits render.Provider (audio-driven); needs one reference image from the avatar bundle rather than a source video.

Synthesis / conclusion

There is no clean, off-the-shelf commercial + local-MPS talking-head engine today. The field splits into: (a) MPS-friendly but non-commercial (Wav2Lip; and SadTalker's bundled components), or (b) cleanly commercial but CUDA/openmmlab-bound (LatentSync, MuseTalk).

Two realistic paths remain, both behind the same render.Provider:

  1. SadTalker on MPS (PIRender) + license diligence — preserves the on-device goal; gated on confirming/eliminating the non-commercial components (Deep3DFaceRecon, Wav2Lip). If diligence fails, SadTalker is dev-only like Wav2Lip.
  2. Remote NVIDIA worker, LatentSync (Apache) — cleanly commercial, high quality, no per-render cloud cost, but not on-device on the Mac.

Recommendation: spike SadTalker+PIRender on MPS to confirm it runs here (fast signal, ~minutes), and in parallel do the license diligence; if the components can't be cleared for commercial use, fall back to the remote-NVIDIA LatentSync path. Decision for RMI-003 remains open pending that.

SadTalker spike — VERIFIED on MPS (runnability), licensing NOT clean

Ran SadTalker (PIRender branch) on the M1 Max against the HeyGen presenter frame + F5-TTS narration.wav.

Aspect Result
Install on arm64 ✅ clean — no openmmlab/mmcv (clears the MuseTalk wall). Deps: torch/torchvision, face_alignment, facexlib, gfpgan, basicsr, librosa 0.9.2
Device ✅ native MPS (pirender branch auto-selects mps); PYTORCH_ENABLE_MPS_FALLBACK=1
Full pipeline ✅ 3DMM extraction → audio2exp → PIRender FaceRender ~8 fps (170 frames in 21 s) on MPS
Output 256×256, 13.6 s talking head (head motion + expression from a single source frame); serviceable at corner size. Artifact: sadtalker_spike_result.mp4
Patches needed setuptools<81 (pkg_resources), basicsr functional_tensor→functional, deprecated np.float/np.int across facexlib+SadTalker, numpy pinned to 1.23.5 (SadTalker's target)
Weight sourcing GitHub-release CDN is blocked in this environment (http:000); all weights pulled from HF mirrors (vinthony/SadTalker-V002rc, leonelhs/facexlib)

Runnability: confirmed — SadTalker is genuinely usable on Apple Silicon; the friction was pip-pinning/old-code numpy compat, not an MPS or dependency wall.

Licensing: NOT clean as-is. The pipeline runs Deep3DFaceReconstruction (the "3DMM Extraction" stage — load-bearing, cannot be skipped) and the safetensors bundles Wav2Lip. SadTalker's own README disclaims: "uses models from Deep3DFaceReconstruction and Wav2Lip; review those licenses separately." Deep3DFaceRecon is tied to the Basel Face Model (academic/non-commercial); Wav2Lip is non-commercial. So a commercial user cannot treat SadTalker output as clean without either legal clearance of those components or replacing the 3DMM stage — the same component-replacement shape as MuseTalk's DWPose.

Net conclusion for RMI-003

Every on-device-capable engine we found needs a load-bearing component replaced to be commercially shippable:

  • MuseTalk → replace openmmlab DWPose (technical blocker on arm64).
  • SadTalker → replace Deep3DFaceRecon 3DMM stage (licensing blocker).

The only no-fork commercial path is LatentSync (Apache) on a remote NVIDIA worker — clean license, high quality, not on-device. The decision is therefore invest in a component-replacement fork (on-device + commercial) vs accept the remote-NVIDIA worker (commercial, not on-device) — a genuine product call, now fully de-risked technically.