Speaker Recognition Benchmark Using the CHiME-5 Corpus

Daniel Garcia-Romero, David Snyder, Shinji Watanabe, Gregory Sell, Alan McCree, Daniel Povey, Sanjeev Khudanpur

Introduction

While speaker recognition—the task of identifying who is speaking—has seen dramatic improvements through deep learning, most existing benchmarks rely on relatively clean data like telephone speech or curated YouTube clips (e.g., VoxCeleb). These datasets often fail to represent the "nightmare scenario" for audio systems: natural human interaction in complex physical spaces.

This research addresses the gap in far-field, multi-speaker environments. In typical home settings, microphones are often distant from the speaker, and multiple individuals frequently converse simultaneously. By repurposing the CHiME-5 corpus, originally intended for speech recognition, the authors provide a standardized framework to measure the impact of noise, distance, and overlapping speech on speaker recognition performance.

Methodology

The researchers established a baseline using a state-of-the-art x-vector DNN system. The process involves:

  • Feature Extraction: Utilizing 30-dimensional MFCCs to feed a deep neural network that extracts "x-vectors"—fixed-length embeddings representing speaker identity.
  • Scoring: Comparing these embeddings via Probabilistic Linear Discriminant Analysis (PLDA) to verify speaker identity.
  • Diarization: Employing Agglomerative Hierarchical Clustering (AHC) to separate overlapping audio into individual speaker streams.

The benchmark is structured around four tasks that vary the difficulty of enrollment and testing:

  1. Core-Core: Clean, annotated speech for both phases.
  2. Assist-Core: Enrollment uses full audio segments (requiring the system to isolate the speaker), while testing uses clean speech.
  3. Core-Multi: Clean enrollment paired with multi-speaker test segments.
  4. Assist-Multi: A "real-world" scenario where both enrollment and testing involve complex, multi-speaker audio.
Alignment of enrollment and test segments across binaural (BIN) and far-field (Uxx) devices.

Dataset and Benchmark Setup

The benchmark is built upon the CHiME-5 corpus, which consists of 20 dinner parties recorded in real homes using both wearable binaural microphones (close-talking) and six Microsoft Kinect devices (far-field). The authors curated a specific subset of 39 speakers from 18 of these parties. To ensure a robust baseline, the x-vector model was pre-trained on the massive VoxCeleb 1 & 2 datasets, encompassing over 7,000 speakers.

Statistical distribution of the curated CHiME-5 subset, including enrollment durations and speech ratios.

Key Results and Insights

The study reveals a significant "Far-Field Gap." The Equal Error Rate (EER) increases nearly 10x when transitioning from wearable microphones (1.2% EER) to room-mounted Kinect arrays (10% to 23% EER).

Key findings include:

  • Diarization Impact: In multi-speaker test conditions, applying diarization reduced the EER from 20.1% to 12.2%. While helpful, a substantial performance gap remains compared to single-speaker conditions.
  • Enrollment Selectivity: Maintaining high performance in noisy environments requires "highly selective" enrollment—using only the cleanest, non-overlapping audio to define a speaker's profile.
  • Device Variability: Performance varies significantly depending on the specific room acoustics and the placement of the far-field device.

Conclusion and Future Directions

The CHiME-5 speaker recognition benchmark highlights the extreme difficulty of far-field, multi-speaker scenarios. Current baselines, which process microphone channels independently, struggle with the degradations inherent in these environments. Future work should explore microphone-array beamforming and advanced speech separation to bridge the performance gap. Furthermore, the marginal gains seen in the "Assist" enrollment condition suggest a need for more sophisticated methods to leverage noisy enrollment data effectively.

Summary updated

BibTeX
@inproceedings{garciaromero2019speaker,
  author = {Daniel Garcia-Romero and David Snyder and Shinji Watanabe and Gregory Sell and Alan McCree and Daniel Povey and Sanjeev Khudanpur},
  title = {{Speaker Recognition Benchmark Using the CHiME-5 Corpus}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2174},
}
Figure