JHU Diarization System Description

Zili Huang, L. Paola García-Perera, Jesús Villalba, Daniel Povey, Najim DehakIberSPEECH 2018 · OpenReview

TL;DR

The JHU team presents their speaker diarization system for the Iberspeech-RTVE challenge, adapting their Kaldi-based pipeline to Spanish broadcast audio. They found that a well-trained Speech Activity Detector (SAD) is crucial, and that x-vector embeddings generally outperform i-vectors, especially when fused.

What Problem Does This Paper Address?

Speaker diarization, often called "who spoke when," is the task of segmenting an audio recording and labeling each segment with the identity of the speaker. This is a fundamental step for many downstream speech processing tasks, like summarization or information retrieval. The challenge here is to adapt a robust speaker diarization system, typically developed on English broadcast news, to a new domain: Spanish broadcast audio from the RTVE dataset. This new dataset presents unique challenges, including a different language and potentially different acoustic conditions. The paper specifically addresses two scenarios: a "closed" condition where training data is limited and specific to the task, and an "open" condition with more diverse training data.

Methods

The JHU system follows a standard speaker diarization pipeline built within the Kaldi toolkit. This pipeline consists of several key stages:

  • Acoustic Feature Extraction: Wide-band (16KHz) Mel-Frequency Cepstral Coefficients (MFCCs) are extracted.
  • Speech Activity Detection (SAD): A crucial component, the authors train a Deep Neural Network (DNN) SAD model. For the closed condition, this DNN is trained on the Albayzin2016 dataset. For the open condition, they use the same DNN SAD.
  • Embedding Extraction: This is where the system branches for the open and closed conditions.
  • Closed Condition: Due to limited data, they primarily rely on i-vectors, a traditional method for speaker representation.
  • Open Condition: They explore four types of embeddings:
  • x-vector-basic: A standard Time Delay Neural Network (TDNN) trained on augmented Voxceleb1 and Voxceleb2.
  • x-vector-factored: A factored TDNN (TDNN-F) architecture, designed to reduce parameters, trained on a mix of datasets including SRE12-micphn and VoxCeleb.
  • i-vector-basic: Standard i-vectors trained on Voxceleb1 and Voxceleb2.
  • BNF-i-vector: i-vectors enhanced with bottleneck features.
  • PLDA (Probabilistic Linear Discriminant Analysis): Used for speaker modeling and scoring. For the new scenario, PLDA models are trained on the Albayzin2016 data.
  • Score Fusion: For the open condition, scores from the four embedding types are fused with equal weights.
  • Clustering: An Agglomerative Hierarchical Clustering (AHC) algorithm is used to produce the final speaker segmentation. This clustering process involves scanning for optimal thresholds to estimate the number of speakers.

Datasets

  • Albayzin2016: Used for training the SAD, Universal Background Model (UBM), i-vector extractor, and PLDA models for the closed condition. It includes Aragon Radio and 3/24 TV channel data.
  • Albayzin2018 (dev2): The primary dataset for development and tuning, containing Spanish broadcast RTVE data.
  • Voxceleb1 & Voxceleb2: Used for training x-vector embeddings, both augmented and unaugmented.
  • SRE12-micphn, MX6-micphn, VoxCeleb, SITW-dev-core: Used for training factored x-vectors.
  • Fisher: Mentioned as a dataset, but noted as not being in 16KHz, requiring special handling for bottleneck DNN training.

Key Insights and Ideas

  • SAD is King (or Queen): The paper strongly emphasizes that the quality of the Speech Activity Detector (SAD) is paramount for overall diarization performance. A well-trained SAD significantly improves results.
  • X-vectors Shine in the Open: For the open condition, x-vector based systems consistently outperformed i-vector based systems, aligning with previous research.
  • TDNN-F for Efficiency and Performance: The factored TDNN (TDNN-F) architecture for x-vectors showed the best performance among the individual embedding types, suggesting its effectiveness in reducing parameters while maintaining or improving accuracy.
  • Fusion Boosts Performance: Fusing the scores from multiple embedding systems in the open condition led to a noticeable reduction in Diarization Error Rate (DER).
  • Data Scarcity Favors i-vectors (in Closed): In the closed condition, where training data is limited, i-vectors proved more suitable than DNN-based x-vectors, which require substantial data to train effectively.
  • Clustering Nuances: The paper notes interesting behavior where clustering with the actual number of speakers sometimes degraded performance for x-vector systems, suggesting i-vectors might require more prior knowledge of speaker counts.

Key Results

  • Open Condition: The best performing system was a fused system achieving a DER of 9.39% on the Albayzin2018 dev2 dataset. The TDNN-F based x-vector alone achieved a DER of 12.12%.
  • Closed Condition: The i-vector system achieved a DER of 24.03% with supervised calibration.
  • SAD Impact: Table 3 shows that the TDNN SAD trained on Albayzin2016 achieved a DER of 13.39% for the basic x-vector system, significantly outperforming an energy-based SAD (21.52%) and a TDNN SAD trained on Librispeech (15.66%).

Limitations and Future Work

  • Overlap Handling: The current system predicts a single speaker label per frame, making it unable to handle overlapping speech. The authors suggest this would require a fundamental change in the prediction mechanism.
  • Speaker Count Estimation: Accurately estimating the number of speakers, especially in challenging scenarios with many speakers, remains an open problem. The supervised calibration and AHC methods still have room for improvement.
  • Re-segmentation: The authors did not include a re-segmentation step due to time constraints, but plan to incorporate it in future work to potentially further improve performance.
  • Data Limitations for X-vectors (Closed): The poor performance of x-vectors in the closed condition is attributed to insufficient training data, even with augmentation.
  • Language Adaptation: While the system was adapted to Spanish, further research into language-specific acoustic modeling could be beneficial.

Summary updated

Abstract

We present the JHU system for Iberspeech-RTVE Speaker Diarization Evaluation. This assessment combines Spanish language and broadcast audio in the same recordings, conditions in which our system has not been tested before. To tackle this problem, the pipeline of our general system, developed entirely in Kaldi, includes an acoustic feature extraction, a SAD, an embedding extractor, a PLDA and a clustering stage. This pipeline was used for both, the open and the closed conditions (described in the evaluation plan). All the proposed solutions use wide-band data (16KHz) and MFCCs as their input. For the closed condition, the system trains a DNN SAD using the Albayzin2016 data. Due to the small amount of data available, the i-vector embedding extraction was the only approach explored for this task. The PLDA training utilizes Albayzin data followed by an Agglomerative Hierarchical Clustering (AHC) to obtain the speaker segmentation. The open condition employs the DNN SAD obtained in the closed condition. Four types of embeddings were extracted, xvector-basic, xvector-factored, i-vector-basic and BNF-i-vector. The x-vector-basic is a TDNN trained on augmented Voxceleb1 and Voxceleb2. The x-vector-factored is a factored-TDNN (F-TDNN) trained on SRE12-micphn, MX6-micphn, VoxCeleb and SITWdev-core. The i-vector-basic was trained on Voxceleb1 and Voxceleb2 data (no augmentation). The BNF-i-vector is a BNF-posterior i-vector trained with the same data as xvector-factored. The PLDA training for the new scenario uses the Albayzin2016 data. The four systems were fused at the score level. Once again, the AHC commputed the final speaker segmentation. We tested our systems in the dev2 data and observed that the SAD is of importance to improve the results. Moreover, we noticed that xvectors were better than i-vectors, as already observed in previous experiments.
BibTeX
@inproceedings{huang2018jhu,
  author = {Zili Huang and L. Paola García-Perera and Jesús Villalba and Daniel Povey and Najim Dehak},
  title = {{JHU Diarization System Description}},
  booktitle = {IberSPEECH},
  year = {2018},
  url = {https://openreview.net/forum?id=gW0iWI541P},
}
Figure