Voice Detection, Turn Detection and Diarization

with PyAnnote, Nvidia NeMo and Pipecat Smart Turn

Turn Detection and Transcription Re-emerge as Important

As voice assistants become more popular, there is a renewed importance for techniques to detect a) when a user stops speaking (so the AI responds in time, but also without interrupting) and b) the presence of multiple speakers.

Both tasks are closely related, so I describe the theory behind both. I then survey common libraries, namely Pyannote and Nvidia NeMo, as well as the new Smart Turn library from Pipecat.

Cheers, Ronan

🛠 Get the Voice Detection Scripts

💡 Consulting (Technical Assistance OR Market Insights)

🤝 Join the Trelis Team / Trelis Developer Colabs

💸 Grants Program

---

Turn Detection and Diarization

Turn detection and diarization serve distinct but related purposes in speech processing. Turn detection identifies when a speaker has finished speaking, while diarization attributes speech segments to specific speakers.

Turn Detection Fundamentals

Turn detection involves two key components:

1. Voice Activity Detection (VAD) to identify speech vs non-speech in 30ms frames

2. Turn completion detection to determine if a speaker has finished

The core challenge is handling:

1. Long pauses during thought

2. Filler words ("um", "and") that don't indicate completion

3. Tonal indicators of completion vs continuation

PipeCat's Smart Turn project addresses this using:

1. A 2.3GB wave2vec-BERT model for audio embeddings

2. Additional classification layers (<1MB) to predict completion

3. Training on both synthetic and human speech data (4000+ samples)

Diarization Architecture

The standard diarization pipeline consists of:

1. Voice activity detection (30ms frames)

2. Speech segmentation (combining VAD frames)

3. Speaker embedding extraction

4. Clustering to identify distinct speakers

Two main approaches exist:

Pyannote Pipeline:

1. Uses dedicated segmentation model instead of VAD

2. Handles up to 3 speakers with overlap detection

3. Employs bidirectional LSTM for context

4. 25MB embedding models

NVIDIA NEMO Pipeline:

1. Uses MarbleNet VAD (smaller/faster than alternatives)

2. Implements multiscale embeddings at different time windows

3. TitaNet embeddings (~100MB)

4. Employs neural refinement for overlap detection (MSDD)

Key Technical Challenges

Performance degrades significantly with:

1. Overlapping speech

2. Background noise

3. Multiple speakers with short utterances

4. Varying speaker tones/styles

Current limitations:

1. Turn detection models are large (2.3GB) and slow

2. Diarization struggles with speaker overlap

3. Short utterances are difficult to attribute

4. Speaker count detection is unreliable without prior knowledge

The field remains active with ongoing work to:

1. Reduce model sizes

2. Improve overlap detection

3. Handle more dynamic speaking scenarios

4. Better integrate with downstream applications

For practical implementations, testing with representative audio samples and potentially combining approaches (e.g., Pyannote segmentation with NEMO speaker detection) may yield better results than any single solution.