FAST Streaming-Text-to-Speech Models

Kyutai TTS vs Whisper

Whisper has long been the go-to text to speech model. Now Voxtral from Mistral is higher quality, and Kyutai’s TTS - with a streaming dual-channel architecture - is faster.

I explain how to run inference locally or to set up a remote Rust based server that can handle simultaneous requests.

Cheers, Ronan

🤖 Purchase ADVANCED-audio Repo Access

---

🗝️ Get Trelis All Access:

1. Access all SEVEN Trelis Github Repos (-robotics, -vision, -evals, -fine-tuning, -inference, -voice, -time-series)

2. Support via Github Issues & Trelis’ Private Discord

3. Early access to Trelis videos via Discord

---

TIMESTAMPS:

0:00 Streaming Speech to Text Demo with Kyutai TTS

0:42 Demo en français

1:05 Video Overview

2:42 Resources & Repo

3:15 Running Kyutai TTS on your Mac

5:15 Run streaming TTS in a notebook

5:58 Word timestamping

8:52 Text and Audio Assisted Transcription

11:46 Fast STREAMING TTS server with Rust

15:27 Streaming vs Whisper TTS vs Voxtral

19:53 Theory of Timestamping

22:55 Whisper vs Kyutai TTS architectures

24:34 How Kyutai is trained (with whisper timestamped data)

25:50 Wrap up

---

Real-Time Speech-to-Text: Comparing Kyutai, Whisper, and Voxtral Architectures

Kyutai's speech-to-text model offers distinct advantages for real-time transcription compared to alternatives like Whisper and Voxtral. The key differences lie in their architectural approaches and how they handle streaming audio.

Core Architectural Differences

Kyutai uses a decoder-only architecture that processes audio in parallel channels:

1. One channel handles incoming audio

2. A second channel generates text with a fixed delay (0.5s for small model, 2s for large model)

3. Built-in voice activation detection for faster end-of-phrase processing

In contrast:

1. Whisper uses an encoder-decoder architecture requiring full audio context

2. Voxtral employs a decoder-only approach but without the fixed delay buffer

Model Specifications

Kyutai offers two models:

1. Small model: ~1B parameters, 0.5s delay

2. Large model: 2.6B parameters, 2s delay

Training data - Pre-trained on 2.5M hours of Whisper-timestamped data

Performance Characteristics

Latency and throughput advantages of Kyutai:

1. Autoregressive generation with fixed delay

2. Automatic word timestamping from delay alignment

3. Voice activation detection reduces end-of-phrase latency

4. Rust server implementation for high-speed processing

Whisper limitations:

1. Must recompute full history for streaming

2. Higher computational overhead due to encoder-decoder architecture

3. Requires attention map analysis for word-level timestamps

Implementation Options

Local inference:

1. Supports CPU-only operation

2. MLX optimization available for Mac

3. PyTorch implementation for broader compatibility

Server deployment:

1. Rust server implementation

2. Exposes TCP port 8080

3. Supports real-time audio streaming

4. Configurable real-time factor for faster processing

Language Support

Current language capabilities:

1. English

2. French

Practical Considerations

Accuracy trade-offs:

1. Quality limited by Whisper-based training data

2. Main advantage is streaming performance, not transcription accuracy

3. Fixed delay provides stability in output compared to Voxtral's immediate transcription

Error correction:

1. Supports text and audio-assisted correction

2. Requires both audio sample and correct text for optimal results

3. Useful for handling domain-specific terminology