End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations

Giovanni Morrone, Samuele Cornell, Luca Serafini, Enrico Zovato, Alessio Brutti, Stefano Squartini

Introduction

Speaker diarization—the process of partitioning an audio stream into homogeneous segments according to speaker identity—is a fundamental task in speech processing. It is critical for downstream applications such as automated transcription of medical consultations or indexing large-scale call center archives. Traditionally, diarization has relied on clustering-based methods: segmenting audio, extracting speaker embeddings (voice fingerprints), and grouping them. However, these methods often fail during overlapped speech, where multiple individuals talk simultaneously. In natural telephone conversations, such overlaps can constitute up to 20% of the duration, leading to significant errors in traditional systems.

While End-to-End Neural Diarization (EEND) models have emerged to handle overlaps, they typically require vast amounts of training data and often suffer from high latency, limiting their utility in real-time scenarios like live captioning. This research proposes an optimized Speech Separation Guided Diarization (SSGD) framework that achieves state-of-the-art performance with ultra-low latency.

Integrated Speech Separation Guided Diarization

The proposed SSGD approach shifts the paradigm from clustering segments to untangling the audio mixture into distinct speaker streams. The pipeline consists of three core components designed for efficiency and low-latency operation.

The proposed Speech Separation Guided Diarization (SSGD) pipeline, featuring framing, separation, leakage removal, and VAD.
  1. Speech Separation (SSep): The system utilizes causal architectures—specifically Conv-TasNet, DPRNN, or DPTNet—to separate the mixed signal into individual speaker channels. By using causal models, the system avoids "looking ahead" into future audio frames, maintaining a minimal algorithmic latency.
  2. Leakage Removal: Separation is rarely perfect; "leakage" often occurs when a faint version of one speaker's voice appears in another's channel. The researchers introduced a computationally efficient algorithm that uses Signal-to-Distortion Ratios (SI-SDR) to identify and suppress these leaked segments by comparing separated streams back to the original mixture.
  3. Voice Activity Detection (VAD): A Temporal Convolutional Network (TCN) is employed to detect speech presence on each separated stream.

The Power of Joint Training

A key innovation of this work is the transition from disjoint training to end-to-end integration. In a disjoint setup, the separator and VAD are trained independently. However, by fine-tuning these modules together, the VAD learns to accommodate the specific artifacts and distortions produced by the separation model.

This joint training acts as a "secret sauce," making the system significantly more robust. Notably, the researchers found that with joint training, the explicit leakage removal algorithm becomes less critical, as the VAD naturally learns to ignore "ghost" voices on the separated channels.

Experimental Results and Performance

The system was evaluated on the Fisher Corpus and the CALLHOME benchmark, focusing on two-speaker telephone conversations. The results indicate that the integrated SSGD model not only matches but often exceeds the performance of much larger, high-latency models.

Diarization performance on the CALLHOME dataset, highlighting the benefits of joint SSep and VAD fine-tuning.

Key findings from the evaluation include:

  • State-of-the-Art Accuracy: On the CALLHOME 2-speaker subset, the model achieved a Diarization Error Rate (DER) as low as 8.4% (Full Evaluation) and 4.1% (Fair Evaluation) when using joint fine-tuning.
  • Low Latency: The system operates with an algorithmic latency of just 0.1 seconds. Experiments showed that increasing lookahead beyond this point yielded diminishing returns, suggesting 0.1s is an optimal sweet spot for telephone tasks.
  • Data Efficiency: The model achieved these results using approximately 1,900 hours of training data—nearly ten times less than the 10,000 hours required by some competing EEND architectures.
  • Computational Efficiency: With a Real-Time Factor (RTF) of 0.16 on a standard desktop CPU, the system is well-suited for real-time deployment.

Conclusion and Future Directions

This work demonstrates that integrating speech separation and VAD into a single, low-latency pipeline provides a powerful solution for diarizing overlapped telephone speech. By providing separated audio streams as a byproduct, the system also facilitates high-quality Automatic Speech Recognition (ASR), reaching word error rates comparable to those achieved with perfect manual separation.

While currently optimized for two-speaker scenarios, future research will focus on scaling this framework to handle an unknown number of speakers in more complex environments, such as multi-person meetings, by combining local separation with global online clustering.

Summary updated

BibTeX
@article{morrone2024end,
  author = {Giovanni Morrone and Samuele Cornell and Luca Serafini and Enrico Zovato and Alessio Brutti and Stefano Squartini},
  title = {{End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations}},
  journal = {Speech Commun.},
  year = {2024},
  doi = {10.1016/j.specom.2024.103081},
}
Figure