SPIKE-022 — MuseTalk on Apple Silicon (commercial engine)¶
RMI: RMI-OACORE-022 · Status: ☑ done (negative result) · Follows: commercial-intent decision
Goal: MuseTalk is MIT-licensed with commercial-usable weights, so it was the leading shippable MPS candidate. Verify whether it actually runs on the M1 Max before building Phase 2 around it.
Result: NOT viable locally on Apple Silicon (as-is)¶
| Check | Finding |
|---|---|
| Licensing | ✅ MIT code; weights "available for any purpose, even commercially" |
tensorflow==2.12.0 pin |
Not an inference blocker — not imported for inference (training/tensorboard only); skippable |
| Device | inference.py hardcodes cuda-or-cpu; patchable to MPS like Wav2Lip |
| DWPose face preprocessing | ❌ Hard-depends on mmpose/mmdet/mmcv (openmmlab). musetalk/utils/preprocessing.py: from mmpose.apis import inference_topdown, init_model |
| mmcv install on arm64 mac | ❌ No arm64 wheel; mim install mmcv falls into a failing source build. openmmlab is the well-known Apple-Silicon blocker |
| macOS support upstream | ❌ Unresolved — issue #393 open, no working solution |
| Weight footprint | Multi-model (musetalk v1.0+v1.5 UNet, sd-vae-ft-mse, whisper-tiny, DWPose, syncnet, face-parse) across several HF repos + gdown |
Getting MuseTalk on the M1 would require forking out DWPose (replacing the openmmlab landmark detector with a mac-friendly one — face_alignment / mediapipe / insightface), then still validating UNet+VAE+whisper MPS-op compatibility. High effort, uncertain, ongoing maintenance burden.
Pivotal conclusion¶
Across the three primary engines, no single engine is both commercially licensed and locally runnable on Apple Silicon:
| Engine | Commercial | Local on M1 (MPS) |
|---|---|---|
| Wav2Lip | ❌ non-commercial | ✅ proven (SPIKE-001) |
| MuseTalk | ✅ MIT | ❌ mmcv/DWPose blocker |
| LatentSync | ✅ Apache-2.0 | ❌ CUDA-only |
Since intent is commercial, the local-on-M1 + commercial combination is not
readily achievable with these engines. Realistic paths (all behind the same
render.Provider):
- Remote NVIDIA worker running LatentSync (Apache, high quality) or MuseTalk — commercial-clean, high quality, uses your own GPU box (not a third-party cloud avatar API). Trades the "on-device on the Mac" goal for the avatar stage while keeping the local-first architecture and zero per-render cloud cost.
- Fork MuseTalk to drop DWPose and attempt MPS — high effort/uncertain.
- Find an alternative commercial + MPS-friendly engine (e.g. SadTalker and other non-openmmlab, non-CUDA-only options) — needs a short research pass.
The provider architecture (proto, server, Go adapter, bundle) is unchanged in all three; only the engine module + where it runs differ. Decision required (RMI-003) before Phase 2 build.
Note¶
Wav2Lip stays a valid non-commercial dev/reference engine (SPIKE-001) — useful to exercise the full stack locally — but must not be shipped given commercial intent.