Skip to content

SPIKE-001 — Engine runnability on Apple Silicon

RMI: RMI-OACORE-001 (also discharges most of RMI-OACORE-002) · Status: ◐ in progress

Goal: determine which local lip-sync/avatar engine runs on the M1 Max dev machine (Apple Silicon, MPS) well enough to render a corner presenter, so RMI-003 can pick the first engine to build. Throwaway Python; installs live in the scratchpad, only findings are recorded here.

Environment (verified)

Item Result
Shell arch x86_64 under Rosetta; arch -arm64 runs native arm64 (as with the voice servers)
Python/venv Reused the arm64 venv from the voice work
PyTorch 2.13.0, torch.backends.mps.is_available() == True — MPS foundation already present
ffmpeg / git present
Disk free ~173 GB
Narration input narration.wav — 16 kHz mono, 13.6 s (from the F5-TTS demo, resampled)

The PyTorch+MPS stack is already installed (it came in as an f5-tts-mlx dependency), so the per-engine cost is just the engine's own deps + weights.

Engine fact sheet (verified against upstream)

Engine Complete render provider? Apple Silicon (MPS) License / commercial use Verdict
Wav2Lip (Rudrabha/Wav2Lip) ✅ lip-sync ✅ Proven this spike (model on MPS, face-detect CPU) ❌ Non-commercial only — "any form of commercial use is strictly prohibited"; LRS2-trained weights Runs, but not shippable commercially. Dev/reference engine only
MuseTalk (TMElyralab/MuseTalk) ✅ lip-sync (real-time) ❓ Unproven — CUDA-oriented setup, no macOS guidance; may port like Wav2Lip ✅ MIT code, weights "available for any purpose, even commercially" Best shippable MPS candidate — needs its own MPS spike
LatentSync (bytedance/LatentSync) ✅ lip-sync (diffusion) ❌ CUDA-only, 8 GB (1.5) / 18 GB (1.6) VRAM ✅ Apache-2.0 Commercial-clean quality target — remote NVIDIA worker or future MPS port
LivePortrait (KwaiVGI/LivePortrait — ideation doc misattributed to "KlingAIResearch") ❌ motion pre-stage only Community MPS runs reported check Phase-2 pre-stage, not MVP
EchoMimic / V3 (antgroup/echomimic*) ✅ end-to-end CUDA-oriented (A100/4090) check Later end-to-end alternative, not M1 MVP

Two corrections carried forward from the ideation doc: (1) LatentSync is not a low-effort M1 target — it is CUDA-only. (2) Licensing is a gating factor the ideation doc never raised: the engine that runs most easily on Apple Silicon (Wav2Lip) is the one we cannot ship commercially. The engine that is cleanly commercial and likely MPS-portable is MuseTalk (MIT), but its Apple Silicon runnability is unproven and needs its own spike. LatentSync is commercial + high-quality but CUDA-only (remote worker).

Implication for RMI-003: separate "engine that proved the architecture" (Wav2Lip) from "engine we ship." See the decision pending in the ROADMAP.

Preliminary recommendation (to confirm in RMI-003)

  • MVP engine: Wav2Lip on MPS. First runnable, lightest install, sufficient for a 180–320 px corner avatar (defects that matter at full-frame are largely invisible at corner size — to be confirmed by the render).
  • LatentSync: deferred, targeted at a remote NVIDIA worker (or a future MPS port); it is the quality benchmark, not the first build.
  • MuseTalk: fallback MPS candidate if Wav2Lip quality is inadequate at corner size.

This keeps providers/latentsync as a planned package but makes providers/wav2lip the first engine implemented — the ROADMAP's <engine> resolves to wav2lip for the MVP unless RMI-003 overrides.

Blocker to finish the run

Completing the actual render (and the human quality judgment the PLAN requires) needs a source face video — a real face clip for the lip-sync model to drive. This is the RMI-004 asset (the reusable idle/presenter clip). The spike deliberately does not grab an arbitrary person's likeness. Options to unblock:

  1. Provide a short (~10–20 s) neutral, mouth-mostly-closed, fixed-camera face clip to use as the spike input and the seed for the john avatar bundle (RMI-004).
  2. Approve using a standard public-domain lip-sync test clip purely to prove the MPS pipeline executes (runnability only; not a deliverable identity).

Wav2Lip run — VERIFIED on Apple Silicon MPS

Source face = a cached HeyGen presenter clip (~/Library/Caches/vac/avatar/…, 1080p head-and-shoulders, front-facing); audio = the F5-TTS narration.wav.

Aspect Result
Device Wav2Lip generator on MPS; s3fd face detection on CPU (its base class rejects non-cpu/cuda devices)
Config source downscaled to 640×360; PYTORCH_ENABLE_MPS_FALLBACK=1
Latency ~4 min for a 13.6 s clip at 360p (start→finish). Face detection (CPU) dominates
Output 640×360, 13.6 s (== narration), H.264+AAC — mouth correctly re-synced to F5-TTS audio
Quality Acceptable at corner size (~280 px): mouth-region blur that shows at full res is largely invisible when composited as a corner presenter
Artifact wav2lip_spike_result.mp4 (copied to repo root of plexusone/ for review)

At 1080p, CPU face detection was ~100–200 s/batch (~40 min projected) — unusable; downscaling the source to 360–480p is essential and harmless for a corner avatar.

Recipe (seeds the future providers/wav2lip server)

  • Python 3.11 arm64 venv (numpy has no 3.14 wheels); numpy<2, torch 2.13 (MPS), librosa==0.10.2, opencv-python, scipy, numba, tqdm.
  • Checkpoints from the camenduru/Wav2Lip HF mirror: wav2lip_gan.pth (415 MB), s3fd.pth (86 MB). Original GDrive links are dead.
  • Patches to Rudrabha/Wav2Lip (all small): (1) device select adds mps; (2) torch.load(..., weights_only=False) in inference.py (2×) and sfd_detector.py; (3) librosa.filters.mel positional→keyword (librosa 0.10); (4) np.int→int in face_detection/utils.py; (5) face detector forced to device='cpu'.
  • Run: model on MPS, face-detect on CPU, source ≤480p, PYTORCH_ENABLE_MPS_FALLBACK=1.

Decision input for RMI-003

Wav2Lip clears the bar as the first engine to build (providers/wav2lip): runs on MPS today, minimal patches, acceptable corner-size quality, lightest deps. LatentSync remains the quality target for a remote NVIDIA worker (CUDA-only); MuseTalk is a later MPS quality-upgrade candidate. Optimizations noted for the implementation phase: face-detect batch/resolution tuning, optional face-region crop to cut detection cost, and caching detected boxes per avatar bundle.

Status of RMI-001 sub-parts

  • [x] Environment validated (arm64 torch + MPS, ffmpeg, disk, narration WAV).
  • [x] Engine runnability researched & upstream repos verified (feeds RMI-002).
  • [x] Preliminary engine recommendation recorded.
  • [x] Actual MPS render verified on the HeyGen source clip + corner-size quality judged acceptable. RMI-001 complete; RMI-003 can select wav2lip as MVP.