Skip to content

OpenAI

OpenAI provides TTS via their text-to-speech API and STT via Whisper.

Features

  • TTS: High-quality voices with natural prosody
  • STT: Whisper model for accurate transcription
  • Languages: 50+ languages supported
  • Streaming: TTS streaming supported

Configuration

import (
    "github.com/plexusone/omnivoice"
    _ "github.com/plexusone/omnivoice/providers/all"
)

// TTS Provider
tts, err := omnivoice.GetTTSProvider("openai",
    omnivoice.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
)

// STT Provider
stt, err := omnivoice.GetSTTProvider("openai",
    omnivoice.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
)

Text-to-Speech

Available Voices

Voice Description
alloy Neutral, balanced
echo Warm, conversational
fable British, storytelling
onyx Deep, authoritative
nova Friendly, upbeat
shimmer Soft, gentle

Basic Usage

result, err := tts.Synthesize(ctx, "Hello, world!", omnivoice.SynthesisConfig{
    VoiceID: "alloy",
})
if err != nil {
    log.Fatal(err)
}

os.WriteFile("output.mp3", result.Audio, 0600)

Models

Model Quality Speed Use Case
tts-1 Good Fast Real-time applications
tts-1-hd Better Slower Pre-generated content 
result, err := tts.Synthesize(ctx, text, omnivoice.SynthesisConfig{
    VoiceID: "alloy",
    Extensions: map[string]any{
        "model": "tts-1-hd",  // Higher quality
    },
})

Output Formats

OutputFormat Audio result.SampleRate
mp3, opus, aac, flac, wav Encoded by OpenAI —
pcm 16-bit PCM 24000
pcm with SampleRate: 16000 Resampled 16-bit PCM 16000
mulaw (or ulaw) mu-law, for phone calls 8000
alaw A-law, for phone calls 8000

OpenAI produces raw speech only as 24 kHz PCM, so the provider converts it for the other raw formats. Names with the rate appended, such as ulaw_8000 and pcm_16000, also work, so a config written for another provider can be reused unchanged.

config := omnivoice.SynthesisConfig{
    VoiceID:      "alloy",
    OutputFormat: "mulaw", // 8 kHz mu-law for Twilio or Telnyx
}

Streamed audio is converted the same way, chunk by chunk.

Streaming

stream, err := tts.SynthesizeStream(ctx, text, omnivoice.SynthesisConfig{
    VoiceID: "alloy",
})
if err != nil {
    log.Fatal(err)
}

for chunk := range stream {
    if chunk.Error != nil {
        log.Printf("Error: %v", chunk.Error)
        break
    }
    playAudio(chunk.Audio)
}

Speech-to-Text

Basic Transcription

result, err := stt.TranscribeFile(ctx, "audio.mp3", omnivoice.TranscriptionConfig{
    Language: "en",
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(result.Text)

With Word Timestamps

result, err := stt.TranscribeFile(ctx, "audio.mp3", omnivoice.TranscriptionConfig{
    Language:             "en",
    EnableWordTimestamps: true,
})

for _, word := range result.Words {
    fmt.Printf("[%.2f-%.2f] %s\n", word.Start, word.End, word.Word)
}

Whisper Models

Model Size Speed Accuracy
whisper-1 Large Medium High 
result, err := stt.TranscribeFile(ctx, "audio.mp3", omnivoice.TranscriptionConfig{
    Model:    "whisper-1",
    Language: "en",
})

Translation

Translate audio to English:

result, err := stt.TranscribeFile(ctx, "french_audio.mp3", omnivoice.TranscriptionConfig{
    Extensions: map[string]any{
        "task": "translate",  // Translate to English
    },
})

Limitations

  • STT Streaming: Not supported (file-based only)
  • File Size: Max 25MB per request
  • Audio Length: Max ~2 hours

Error Handling

result, err := tts.Synthesize(ctx, text, config)
if err != nil {
    switch {
    case strings.Contains(err.Error(), "rate_limit"):
        log.Println("Rate limited, retry with backoff")
    case strings.Contains(err.Error(), "invalid_api_key"):
        log.Println("Check OPENAI_API_KEY")
    default:
        log.Printf("Error: %v", err)
    }
}

Best Practices

  1. Use tts-1 for real-time - Lower latency than tts-1-hd
  2. Batch transcriptions - OpenAI STT is file-based, not streaming
  3. Set language explicitly - Improves accuracy for STT
  4. Use opus for streaming - Smaller chunks, lower latency

Pricing

Service Model Price
TTS tts-1 $15/1M chars
TTS tts-1-hd $30/1M chars
STT whisper-1 $0.006/min

Prices as of 2024. Check OpenAI Pricing for current rates.

Next Steps