Phonetically-Aware Embeddings, Wide Residual Networks with Time-Delay Neural Networks and Self Attention Models for the 2018 NIST Speaker Recognition Evaluation

Ignacio Viñals, Dayana Ribas, Victoria Mingote, Jorge Llombart, Pablo Gimeno, Antonio Miguel, Alfonso Ortega Giménez, Eduardo LleidaInterspeech 2019 · DOI

Introduction

Automatic speaker verification (ASV) systems traditionally treat speech as a generic acoustic signal. However, the phonetic content—the specific sounds being articulated—is intrinsically linked to a speaker's identity. This paper investigates the integration of phonetic awareness into speaker recognition systems to enhance robustness, particularly in the challenging "in-the-wild" conditions of the 2018 NIST Speaker Recognition Evaluation (SRE18). SRE18 introduced significant hurdles, including Audio from Video (AfV) recorded in unconstrained environments and a language mismatch involving Tunisian Arabic.

Methodology

The researchers proposed two primary frameworks for incorporating phonetic knowledge into speaker embeddings:

1. Phonetic i-vectors

In the classical i-vector paradigm, the authors replaced the generic Universal Background Model (UBM) with a phoneme-dependent UBM. By utilizing a system that identifies the phoneme being spoken at each frame, the extraction of speaker characteristics is guided by the underlying phonetic structure, leading to more precise identity modeling.

2. WRN-TDNN with Multi-head Self-Attention

The modern deep-learning approach utilizes a Wide Residual Network (WRN) combined with Time-Delay Neural Networks (TDNN).

  • Phonetic Conditioning: A dedicated WRN-TDNN model is trained for phoneme recognition. The outputs of this model, referred to as phonetic embeddings, are used to condition the feature extraction process of the primary speaker embedding network.
  • Temporal Aggregation: To transform variable-length sequences into fixed-length speaker fingerprints, a multi-head self-attention mechanism is employed. This allows the model to focus on the most informative segments of an utterance while ignoring noise or silence.
Architecture of the phonetically-conditioned WRN-TDNN with multi-head attention.

Experimental Setup

Models were trained on extensive legacy datasets, including SRE 2004-2010, Switchboard, and Fisher. Evaluation was conducted on two NIST SRE18 subsets:

  • CMN2: Tunisian Arabic conversational telephone speech.
  • VAST: Multilingual, noisy audio extracted from YouTube videos. Unlabeled SRE18 data was used for domain adaptation to align the models with the specific acoustic environments of the test sets.

Results and Discussion

The inclusion of phonetic information yielded consistent improvements across different architectures. The phonetic i-vector system achieved an 8% relative improvement over its traditional counterpart.

Performance comparison of systems with and without phonetic information on NIST SRE18 datasets.

As shown in the results table, the impact of phonetic information is particularly evident in the "actual cost" (actC) metric for the VAST dataset, which dropped from 0.724 to 0.657. While x-vector systems remained highly competitive on clean telephone speech (CMN2), the phonetically-aware i-vector approach demonstrated superior resilience in the unconstrained VAST environment. The final "Primary" system, a fusion of these approaches, achieved an EER of 7.63% on CMN2 and 14.60% on VAST.

Conclusion

Explicitly modeling phonetic content serves as a "force multiplier" for speaker recognition, especially in adverse acoustic conditions. The study demonstrates that multi-head attention effectively aggregates informative speech segments into robust "supervectors." Future work will explore the end-to-end integration of phonetic and speaker networks to further optimize performance and simplify the training pipeline.

Summary updated

BibTeX
@inproceedings{vinals2019phonetically,
  author = {Ignacio Viñals and Dayana Ribas and Victoria Mingote and Jorge Llombart and Pablo Gimeno and Antonio Miguel and Alfonso Ortega Giménez and Eduardo Lleida},
  title = {{Phonetically-Aware Embeddings, Wide Residual Networks with Time-Delay Neural Networks and Self Attention Models for the 2018 NIST Speaker Recognition Evaluation}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2417},
}
Figure