Pyannote.Audio: Neural Building Blocks for Speaker Diarization

Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, Marie-Philippe GillICASSP 2020 · DOI

TL;DR

This paper introduces pyannote.audio, an open-source Python toolkit for speaker diarization that leverages PyTorch and provides a suite of trainable, end-to-end neural building blocks. It aims to simplify the creation and optimization of speaker diarization pipelines by offering pre-trained models for key tasks, achieving state-of-the-art performance on several benchmarks.

What Problem Does This Paper Address?

Speaker diarization, the task of identifying "who spoke when" in an audio recording, is a crucial component in many audio analysis applications. Traditional approaches often involve assembling a pipeline of distinct modules, each tackling a specific sub-problem like voice activity detection, speaker change detection, or clustering. This modularity, while flexible, can lead to suboptimal performance due to the lack of joint optimization across the entire pipeline. Furthermore, building and training these pipelines can be complex and require significant expertise. pyannote.audio aims to address this by providing a unified, PyTorch-based framework with pre-trained neural building blocks that can be easily combined and jointly optimized, democratizing access to high-performance speaker diarization.

Methods

pyannote.audio is built around the concept of "neural building blocks" that can be trained and combined to form a diarization pipeline. The core idea is to frame most sub-tasks as sequence labeling problems, where a neural network learns to map a sequence of audio features to a corresponding sequence of labels.

Key methodological contributions include:

  • End-to-End Neural Building Blocks: The toolkit offers trainable neural network implementations for voice activity detection (VAD), speaker change detection (SCD), overlapped speech detection (OSD), and speaker embedding. These blocks can be trained from raw waveforms or handcrafted features.
  • Generic PyanNet Architecture: A flexible, end-to-end neural network architecture (PyanNet) is proposed, which can be adapted for various sequence labeling tasks by adjusting its output layer and pooling strategy.
  • On-the-Fly Data Augmentation: For training, pyannote.audio employs fixed-length sub-sequences and implements on-the-fly data augmentation (e.g., adding noise to random audio chunks). This approach generates a virtually infinite number of augmented samples, improving training robustness and efficiency.
  • Unified Framework for Sequence Labeling: A consistent framework is used across different sequence labeling tasks, simplifying development and integration.
  • Joint Pipeline Optimization: The toolkit supports joint optimization of hyper-parameters across different building blocks within a pipeline, aiming to minimize the overall diarization error rate.
  • Metric Learning for Speaker Embedding: Instead of relying solely on traditional methods like PLDA for speaker embedding, pyannote.audio incorporates metric learning approaches that directly optimize for cosine distance, simplifying the clustering step.

Datasets

The paper mentions the use of several datasets for training and evaluation:

  • AMI: Used for training and evaluation of speaker change detection and speaker diarization pipelines.
  • DIHARD: Used for training and evaluation of voice activity detection and speaker diarization pipelines.
  • ETAPE: Used for training and evaluation of voice activity detection and speaker diarization pipelines.
  • VoxCeleb: Used for training and evaluation of speaker embedding models (specifically for speaker verification tasks).
  • MUSAN: Used for additive noise augmentation during training.

The paper emphasizes that the models are trained and tuned on diverse datasets to ensure broad applicability.

Key Insights and Ideas

  • Democratizing Diarization: By providing pre-trained, state-of-the-art neural building blocks and a unified framework, pyannote.audio significantly lowers the barrier to entry for researchers and developers working on speaker diarization.
  • The Power of Joint Optimization: The paper highlights that jointly optimizing pipeline hyper-parameters, rather than tuning individual modules independently, leads to improved diarization performance.
  • On-the-Fly Augmentation Efficiency: Generating augmented data dynamically during training is shown to be an effective strategy for increasing training sample variability and improving model generalization.
  • End-to-End Waveform Processing: The toolkit's ability to process raw audio waveforms directly in an end-to-end fashion for tasks like VAD and SCD is a significant step towards simplifying feature engineering.

Key Results

pyannote.audio achieves state-of-the-art performance on several tasks and datasets:

  • Voice Activity Detection (VAD): Pre-trained models achieve state-of-the-art performance on various datasets.
  • Speaker Change Detection (SCD): Pre-trained models achieve state-of-the-art performance on various datasets.
  • Overlapped Speech Detection (OSD): Pre-trained models achieve state-of-the-art performance on various datasets.
  • Speaker Diarization Pipelines: The jointly optimized pipeline introduced in [15] achieves competitive results, with improvements noted when incorporating overlapped speech detection. For instance, the pipeline with overlap detection achieved a diarization error rate of 24.8% on AMI and 24.7% on DIHARD.
  • Speaker Embedding: While pre-trained end-to-end models may not reach state-of-the-art on VoxCeleb speaker verification, they demonstrate state-of-the-art performance for some speaker diarization test sets.

Limitations and Future Work

  • Speaker Embedding Performance: The authors acknowledge that their current end-to-end speaker embedding models do not yet reach state-of-the-art performance on the VoxCeleb speaker verification task, though they are competitive for diarization. Future work will likely focus on improving these models.
  • Re-segmentation Unsupervised Nature: While re-segmentation is presented as an unsupervised task, it relies on the output of a diarization pipeline. The performance of re-segmentation is thus dependent on the quality of the initial diarization.
  • Ongoing Research: Several metric learning approaches for speaker embedding are marked as "WIP" (Work In Progress), indicating that further development and evaluation are ongoing.

The paper strongly emphasizes the open-source nature of pyannote.audio and encourages community contributions and further research building upon its foundation.

Summary updated

BibTeX
@inproceedings{bredin2020pyannote,
  author = {Hervé Bredin and Ruiqing Yin and Juan Manuel Coria and Gregory Gelly and Pavel Korshunov and Marvin Lavechin and Diego Fustes and Hadrien Titeux and Wassim Bouaziz and Marie-Philippe Gill},
  title = {{Pyannote.Audio: Neural Building Blocks for Speaker Diarization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9052974},
}
Figure