Advances in Speaker Recognition for Multilingual Conversational Telephone Speech: The JHU-MIT System for NIST SRE20 CTS Challenge

Jesús Villalba, Bengt J. Borgstrom, Saurabh Kataria, Jaejin Cho, Pedro A. Torres-Carrasquillo, Najim DehakOdyssey 2022 · OpenReview

TL;DR

This paper presents the JHU-MIT system for the NIST SRE20 CTS challenge, which focused on multilingual conversational telephone speech with unknown languages. The team explored various neural network architectures for x-vector extraction and developed novel PLDA back-ends (mixture and k-NN) to handle the multilingual nature of the data, achieving competitive results.

What Problem Does This Paper Address?

The core challenge addressed is robust speaker recognition in a multilingual conversational telephone speech setting where the languages are unknown to the system. Traditional speaker recognition systems often rely on language-specific models or assume a limited set of languages. The NIST SRE20 CTS (Conversational Telephone Speech) challenge specifically aimed to push the boundaries by presenting participants with data containing an unspecified mix of languages, forcing the development of systems that are inherently language-agnostic or can adapt to unseen languages. This requires not only effective feature extraction but also back-end scoring mechanisms that can generalize across diverse linguistic and channel conditions.

Methods

The JHU-MIT system employed a two-stage approach:

  1. X-vector Extraction: The researchers experimented with several deep learning architectures for extracting speaker embeddings (x-vectors). These included:
  2. ResNet34: A standard residual network.
  3. Squeeze-Excitation ResNets (SE-ResNets): Architectures that incorporate attention mechanisms to re-weight channel-wise features.
  4. Transformers: Models leveraging self-attention mechanisms, adapted with local attention to manage computational complexity.
  5. EfficientNets: Scaled convolutional networks known for their efficiency.
  6. Novel Architectures: They also explored variations like ResNet34 with Instance Normalization (ResNet34-IN) and a temporal squeeze-excitation (TSE) variant of ResNet34.
  7. PLDA Back-ends: To handle the multilingual nature of the data, they proposed two novel Probabilistic Linear Discriminant Analysis (PLDA) back-end approaches:
  8. PLDA Mixture: This approach assumes that different mixture components of the PLDA model can capture distinct language families, allowing for better modeling of diverse linguistic characteristics.
  9. k-Nearest Neighbor (k-NN) PLDA: This method trains a separate PLDA model adapted to each enrollment speaker by using the nearest speakers (identified through cosine similarity) from the training data. The intuition is that in the local vicinity of a target speaker's embedding, the manifold can be approximated linearly, allowing for more personalized adaptation.

Datasets

The researchers utilized a comprehensive set of datasets for training and development:

Training Datasets: * Switchboard (phase 1-3 and cellular 1-2) * NIST SRE04-12 telephone data * MIXER6 telephone phonecalls * NIST SRE16 Dev (Mandarin, Cebuano) * NIST SRE16 Eval 60% (Cantonese, Tagalog) * NIST SRE18 Dev/Eval (Tunisian Arabic) * Fisher Spanish * VoxCeleb 1+2 (augmented with GSM and AMR-NB telephone codecs) * NIST LRE (downsampled)

They trained x-vectors on a large combination of these datasets (excluding LRE) totaling 304k recordings from 13466 speakers, with on-the-fly data augmentation. PLDA back-ends were trained on NIST SRE04-18 and Fisher Spanish.

Development Datasets: * NIST SRE16 Eval YUE/TGL40% * NIST SRE19 Eval * NIST SRE16-19 dev sets and SRE20 progress/eval sets were used for evaluating individual systems and fusions.

Key Insights and Ideas

  • Robustness over Complexity: Surprisingly, the simpler ResNet34 architecture proved more robust in the challenging multilingual SRE20 CTS scenario compared to more complex models like SE-ResNets and Transformers, which tended to overfit to the training data, particularly English.
  • Instance Normalization Benefits: Replacing Batch Normalization with Instance Normalization in ResNet34 (ResNet34-IN) consistently improved performance across development datasets, suggesting better generalization.
  • Channel-wise Attention: The TSE-ResNet34 architecture, incorporating temporal squeeze-excitation and channel-wise attention pooling, offered significant improvements over standard statistics pooling, especially on development sets, and slightly improved the SRE20 progress set.
  • k-NN PLDA Effectiveness: The proposed k-NN PLDA back-end demonstrated substantial improvements in performance and calibration, particularly by adapting to individual enrollment speakers, which is crucial in multilingual and diverse channel conditions.
  • Data Augmentation Impact: While extensive data augmentation was used for x-vector training, adding more multilingual datasets to x-vector and PLDA training did not consistently improve results on the SRE20 progress set, suggesting that the existing in-domain and augmented data were already quite effective.

Key Results

  • The best single system achieved an Actual Cprimary (Cp) of 0.110 on the SRE20 progress set.
  • The best fusion system obtained an Actual Cp of 0.110 on the progress set (8% better than the single system) and 0.087 on the evaluation set.
  • The k-NN PLDA back-end improved Act. Cp by 68% in SRE16-19 and 22% in SRE20 Progress compared to a single adapted PLDA back-end.
  • The kNN-PLDA-v3 back-end achieved almost perfect calibration on averaged SRE16-19 and SRE20 progress sets.

Limitations and Future Work

  • Overfitting of Complex Models: The paper highlights that more complex architectures like SE-ResNets and Transformers, while performing well on in-domain tasks (e.g., VoxCeleb), overfitted to the training data in the SRE20 CTS challenge, particularly to English. This suggests a need for more robust regularization or architectural choices for out-of-domain and multilingual scenarios.
  • Limited Benefit from Additional Multilingual Data: The study found that incorporating additional unlabeled multilingual datasets (like IARPA Babel, LRE17, Mozilla Common-Voice, etc.) into x-vector and PLDA training did not consistently improve performance on the SRE20 progress set. This might indicate that the existing training data, combined with augmentation, was already sufficient, or that the specific nature of these additional datasets did not align well with the SRE20 CTS challenge.
  • Future Work: The authors mention that their recipes for the JHU systems are publicly available in the Hyperion toolkit, which facilitates further research and development in this area. The success of the k-NN PLDA and PLDA mixture models suggests that exploring more sophisticated adaptation and mixture strategies for handling unknown languages and channels remains a promising direction.

Summary updated

Abstract

We present a condensed description of the joint effort of JHU-CLSP/HLTCOE and MIT-LL for NIST SRE20. NIST SRE20 CTS consisted of multilingual conversational telephone speech. The set of languages included in the evaluation was not provided, encouraging the participants to develop systems robust to any language. We evaluated x-vector architectures based on ResNet, squeeze-excitation ResNets, Transformers and EfficientNets. Though squeeze-excitation ResNets and Efficient-Nets provide superior performance in in-domain tasks like VoxCeleb, regular ResNet34 was more robust in the challenge scenario. On the contrary, squeeze-excitation networks over-fitted to the training data, mostly in English. We also proposed a novel PLDA mixture and k-NN PLDA back-ends to handle the multilingual trials. The former clusters the x-vector space expecting that each cluster will correspond to a language family. The latter trains a PLDA model adapted to each enrollment speaker using the nearest speakers–i.e., those with similar language/channel. The k-NN back-end improved Act. Cprimary (Cp) by 68% in SRE16-19 and 22% in SRE20 Progress w.r.t. a single adapted PLDA back-end. Our best single system achieved Act. Cp=0.110 in SRE20 progress. Meanwhile, our best fusion obtained Act. Cp=0.110 in the progress–8% better than single– and Cp=0.087 in the eval set.
BibTeX
@inproceedings{villalba2022advances,
  author = {Jesús Villalba and Bengt J. Borgstrom and Saurabh Kataria and Jaejin Cho and Pedro A. Torres-Carrasquillo and Najim Dehak},
  title = {{Advances in Speaker Recognition for Multilingual Conversational Telephone Speech: The JHU-MIT System for NIST SRE20 CTS Challenge}},
  booktitle = {Odyssey},
  year = {2022},
  url = {https://openreview.net/forum?id=ctnatFmLpw},
}
Figure