Skip to content

v0.4.0

Release Date: 2026-07-27

Introduces the local avatar render provider infrastructure — audio-driven talking-head video generation on Apple Silicon, behind the same render.Provider interface as cloud providers. The first engine is LivePortrait + JoyVASA (MIT/Apache, fully-free).

Highlights

  • Local render provider: providers/liveportrait-joyvasa — on-device talking-head video from narration audio, no cloud API required
  • Avatar bundle format: avatar/bundle.go — standardized on-disk structure for local avatar assets (metadata + idle clips + reference frames)
  • LocalRender gRPC service: proto/localrender/v1/localrender.proto — shared contract for all local render engines

Added

  • avatar.Bundle, avatar.Metadata, avatar.Resolution types for local avatar bundles; avatar.Load, avatar.LoadFrom, avatar.List, avatar.ListFrom functions
  • proto/localrender/v1/localrender.proto with LocalRender gRPC service: Generate, Status, Download, ListAvatars, Health, LoadModel, UnloadModel, RuntimeInfo
  • providers/liveportrait-joyvasa/provider.go implementing render.Provider, render.AudioUploader, and io.Closer
  • providers/liveportrait-joyvasa/server/ Python gRPC server with inference pipeline (HuBERT, DiT motion generator, LivePortrait renderer, mediapipe face analyzer, ffmpeg encoding)
  • registry.RenderProviderFactory via NewFromConfig() for the local provider

Architecture

The local provider mirrors the cloud provider pattern:

Go Client (providers/liveportrait-joyvasa/*.go)
    │ render.Provider: Generate/Status/Download
    │
    │ gRPC over Unix Socket
    ▼
Python Server (server/joyvasa_server.py)
    │ LocalRender gRPC service
    │
    ▼
Inference Pipeline
    audio → HuBERT → DiT motion → LivePortrait render → ffmpeg encode

Performance

On Apple Silicon (M-series):

Metric Value
Resolution 512×512
Speed ~5 min for 13.7s output
Memory ~4GB with model loaded

Conv3D and grid_sampler_3d fall back to CPU on MPS (torch 2.3). torch 2.5+ may add native MPS support for significant speedup.

Commercial Licensing

Both LivePortrait and JoyVASA are MIT-licensed. The provider uses mediapipe (Apache-2.0) for face detection, replacing the non-commercial InsightFace models used in the upstream repos.

Full Changelog | Compare