On the Usage of Phonetic Information for Text-Independent Speaker Embedding Extraction

Shuai Wang, Johan Rohdin, Lukás Burget, Oldrich Plchot, Yanmin Qian, Kai Yu, Jan Cernocký

Abstract

In text-independent speaker verification, the goal is to extract speaker-specific features while ignoring linguistic content. This paper investigates the paradoxical role of phonetic information in this process. We demonstrate that while phonetic details enhance frame-level feature extraction through multitask learning, they introduce unwanted session variability at the segment level. By applying adversarial training to suppress phonetic information in the final embedding, we achieve a significant performance boost, reaching a 3.17% Equal Error Rate (EER) on the VoxCeleb dataset.

Introduction

The core challenge of text-independent speaker recognition is isolating the "who" from the "what." An ideal speaker embedding should be a pure numerical representation of a voice, invariant to the specific words spoken. However, phonetic content often acts as a confounding factor, leading to session variability that degrades system performance.

Interestingly, prior research suggests that incorporating phonetic information can improve speaker recognition. This study resolves this contradiction by pinpointing where phonetic information aids the model and where it hinders it. The authors hypothesize that phonetic awareness is crucial for low-level acoustic feature extraction but should be discarded when forming the final speaker identity representation.

Methodology

The research builds upon the standard x-vector architecture, which utilizes a Time-Delay Neural Network (TDNN) for frame-level processing followed by statistics pooling to generate a segment-level embedding. Two primary strategies are explored:

  1. Multitask Learning (MT): The network is jointly trained to recognize both the speaker and the phonemes, encouraging the retention of phonetic features.
  2. Adversarial Training (ADV): A Gradient Reversal Layer (GRL) is employed to ensure the network learns speaker-discriminative features that are simultaneously uninformative for phoneme classification.

These strategies are applied at two distinct stages: frame-level and segment-level.

Frame-level Phonetic Information

At the frame level, the model processes short acoustic windows. The authors test whether encouraging phonetic awareness here helps the TDNN extractor $M_f$ learn better representations.

Architecture for frame-level phonetic classification.

Segment-level Phonetic Information

At the segment level, after statistics pooling, the goal is to create a robust embedding. Here, the authors test if suppressing phonetic information (ADV) leads to a "cleaner" speaker representation.

Architecture for segment-level phonetic classification.

Experimental Setup

The models were trained using the VoxCeleb1 and VoxCeleb2 development sets, comprising over 2 million utterances. Evaluation was performed on the standard VoxCeleb1 test set. Since manual transcriptions were unavailable, "silver standard" phoneme labels were generated using a Kaldi-based recognizer trained on the TED-LIUM corpus.

Results and Discussion

The experiments confirm the "Early Help, Late Harm" hypothesis. Multitask learning at the frame level (FRM-MT) improves the baseline, while adversarial training at the same stage (FRM-ADV) is detrimental (EER 5.24%), as it strips away necessary acoustic patterns. Conversely, adversarial training at the segment level (SEG-ADV) successfully removes linguistic noise from the final embedding.

The best results were achieved by combining these approaches: using multitask learning to optimize the feature extractor and adversarial training to refine the final embedding.

System Equal Error Rate (EER)
x-vector baseline 3.73%
Frame-level Multitask (FRM-MT) 3.38%
Segment-level Adversarial (SEG-ADV) 3.35%
COMBINE (FRM-MT + SEG-ADV) 3.17%

Conclusion

This work demonstrates that phonetic information is a vital but temporary tool in speaker embedding extraction. Strategic use of multitask and adversarial training allows models to benefit from phonetic context during feature extraction while ensuring the final embedding remains focused solely on speaker identity. Future work will explore more granular phonetic units and improved label generation to further enhance these "content-pure" representations.

Summary updated

Abstract

Embeddings extracted by deep neural networks have become the state-of-the-art utterance representation in speaker recognition systems. It has recently been shown that incorporating frame-level phonetic information in the embedding extractor can improve the speaker recognition performance. On the other hand, in the final embedding, phonetic information is just an additional source of session variability which may be harmful to the text-independent speaker recognition task. This suggests that at the embedding level phonetic information should be suppressed rather than encouraged. To verify this hypothesis, we perform several experiments that encourage or/and suppress phonetic information at various stages in the network. Our experiments confirm that multitask learning is beneficial if it is applied at the frame-level stage of the network, whereas adversarial training is beneficial if it is used at the segment-level stage of the network. Additionally, the combination of these two approaches improves the performance further, resulting in an equal error rate of 3.17% on the VoxCeleb dataset.
BibTeX
@inproceedings{wang2019usage,
  author = {Shuai Wang and Johan Rohdin and Lukás Burget and Oldrich Plchot and Yanmin Qian and Kai Yu and Jan Cernocký},
  title = {{On the Usage of Phonetic Information for Text-Independent Speaker Embedding Extraction}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-3036},
}
Figure