Local Avatar Render¶

This guide covers the local avatar render provider — on-device talking-head video generation using the same render.Provider interface as cloud providers.

Overview¶

The local render system generates audio-driven talking-head videos on Apple Silicon without requiring a cloud API. It uses:

LivePortrait (MIT) — portrait animation renderer
JoyVASA (MIT) — audio-to-motion diffusion stage
mediapipe (Apache-2.0) — face detection (commercial-clean)

Architecture¶

┌─────────────────────────────────────────────────────────────┐
│ Go Client (providers/liveportrait-joyvasa/provider.go)      │
│   render.Provider: Generate/Status/Download                 │
└─────────────────────────────────────────────────────────────┘
                              │
                              │ gRPC over Unix Socket
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Python Server (server/joyvasa_server.py)                    │
│   LocalRender gRPC service                                  │
│   - async job queue                                         │
│   - model lifecycle (LoadModel/UnloadModel)                 │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Inference Pipeline (server/inference.py)                    │
│   1. Audio → HuBERT features                                │
│   2. HuBERT → diffusion motion generator (DiT)              │
│   3. Motion → LivePortrait warp+render                      │
│   4. Frames → ffmpeg H.264 encode                           │
└─────────────────────────────────────────────────────────────┘

Setup¶

1. Start the Python Server¶

cd providers/liveportrait-joyvasa/server

# Create venv (use arch -arm64 on Apple Silicon)
arch -arm64 python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Generate proto stubs
./generate_proto.sh

# Start server
./run.sh              # Model loaded on first request
./run.sh --auto-load  # Pre-load model at startup

The server listens on /tmp/omniavatar-liveportrait-joyvasa.sock.

2. Use the Go Provider¶

import (
    "github.com/plexusone/omniavatar-core/render"
    lp "github.com/plexusone/omniavatar-core/providers/liveportrait-joyvasa"
)

// Create provider
provider, err := lp.New("")  // uses default socket

// Upload local audio (returns local:// URL)
audioURL, err := provider.UploadAudio(ctx, "narration.wav", audioFile)

// Generate video
job, err := provider.Generate(ctx, render.GenerateRequest{
    AvatarID: "john",          // avatar bundle name
    AudioURL: audioURL,
    Extensions: map[string]any{
        "seed":         int64(42),    // deterministic output
        "motion_scale": float32(1.2), // adjust expressiveness
    },
})

// Wait for completion
status, err := render.Wait(ctx, provider, job.ID, 3*time.Second)

// Download result
f, _ := os.Create("output.mp4")
err = provider.Download(ctx, job.ID, f)

Avatar Bundles¶

Avatar bundles are on-disk directories containing the source material for rendering. Default location: ~/.omniavatar/avatars/.

Bundle Structure¶

~/.omniavatar/avatars/
    john/
        metadata.json        # bundle metadata
        idle/
            idle.mp4         # neutral, mouth-closed clip
        references/          # optional still frames
            front.png

metadata.json¶

{
  "name": "john",
  "description": "John's avatar",
  "fps": 25,
  "resolution": {
    "width": 512,
    "height": 512
  },
  "created": "2026-07-27",
  "capture_guidance": "See docs/specs/local-render/TRD.md"
}

Loading Bundles¶

import "github.com/plexusone/omniavatar-core/avatar"

// List available bundles
names, err := avatar.List()

// Load a specific bundle
bundle, err := avatar.Load("john")
fmt.Println(bundle.Metadata.Name)       // "john"
fmt.Println(bundle.PrimaryIdleClip())   // full path to idle clip

Environment Variables¶

Variable	Description	Default
`OMNIAVATAR_BUNDLES_DIR`	Avatar bundles directory	`~/.omniavatar/avatars`
`PYTORCH_ENABLE_MPS_FALLBACK`	Enable CPU fallback for unsupported MPS ops	`1` (set by run.sh)

Performance¶

On Apple Silicon (M-series):

Metric	Value
Resolution	512×512
Speed	~5 min for 13.7s output
Memory	~4GB with model loaded

The bottleneck is Conv3D and grid_sampler_3d falling back to CPU. torch 2.5+ may add native MPS support for significant speedup.

Troubleshooting¶

Server Not Responding¶

Check if the server is running:

ls -la /tmp/omniavatar-liveportrait-joyvasa.sock

Model Loading Fails¶

Ensure PYTORCH_ENABLE_MPS_FALLBACK=1 is set (run.sh sets this automatically).

Out of Memory¶

The model requires ~4GB. Close other memory-intensive applications or use UnloadModel when not actively rendering.