A Domain Robust Pre-Training Method with Local Prototypes for Speaker Verification

Qing Gu, Yan Song, Haoyu Song, Nan Jiang, Lirong Dai, Ian McLoughlinInterspeech 2025

TL;DR

This paper introduces a novel pre-training method for speaker verification (SV) that significantly improves robustness to domain shifts by leveraging local prototypes and a domain-aware alignment strategy. By learning domain-invariant local features, the method achieves state-of-the-art performance on challenging benchmarks, addressing a key limitation of existing self-supervised SV techniques.

What Problem Does This Paper Address?

Speaker verification (SV) systems aim to confirm a person's identity based on their voice. While deep learning has greatly advanced SV, current methods often struggle with "domain shifts" – variations in speech caused by different recording conditions, languages, speaking styles, or emotions. These shifts can degrade performance because models trained on one type of data may not generalize well to another. Existing self-supervised learning (SSL) methods, while reducing reliance on labeled data, also tend to be sensitive to these domain variations. This paper tackles the challenge of building more robust SV systems that can maintain high accuracy across diverse acoustic environments and speaking conditions.

Methods

The proposed method consists of two main stages:

  1. Domain-Agnostic Pre-Training: This stage uses a transformer-based Siamese architecture with a self-distillation framework. A "student" branch learns to reconstruct masked local speech features, guided by a "teacher" branch updated via exponential moving average (EMA). Crucially, this stage also learns "local prototypes" through online clustering. These prototypes act as a universal representation of frame-level speech characteristics, capturing latent semantic information.
  2. Domain-Aware Alignment: This stage builds upon the learned prototypes. It employs an episodic training procedure, where data from different domains is paired. Using a technique inspired by WBDA (Within- and Between-class Distribution Alignment), the method aligns frame-level feature distributions across these domains. This alignment explicitly aims to remove speaker-irrelevant variations and learn domain-invariant embeddings, making the system more robust to domain shifts.

After pre-training, the learned features are fine-tuned using a standard ECAPA-TDNN model with utterance-level supervision.

Datasets

  • Training: CNCeleb dataset was used for pre-training.
  • Evaluation:
    • CNCeleb (CN-E evaluation set)
    • VoxCeleb1 original test dataset (Vox1-O)
    • VoxCeleb2 dataset was used for fine-tuning.

The CNCeleb dataset is noted for its multi-genre and cross-genre challenges, making it suitable for testing domain robustness. VoxCeleb2 is a large-scale dataset commonly used for SV research.

Key Insights and Ideas

  • Local Prototypes as a Universal Model: The paper highlights that learning local prototypes from unlabeled data can create a powerful, domain-agnostic representation of frame-level speech features. These prototypes act like a "universal model" for speech characteristics.
  • Frame-Level Domain Alignment is Key: Instead of focusing solely on utterance-level features, the method's success stems from aligning domain characteristics at the finer frame level. This is more effective for handling subtle variations that contribute to domain shifts.
  • Self-Distillation Enhances Feature Learning: The use of a self-distillation framework with a transformer backbone allows for effective learning of rich, contextual local features without explicit labels.
  • Episodic Training for Domain Generalization: By simulating domain adaptation tasks within an episodic training framework, the model learns to generalize better to unseen domains.

Key Results

  • The proposed method (TE w/ PT) significantly outperforms the baseline Transformer Encoder without pre-training (TE w/o PT) and even the raw FBank features on the CN-E evaluation set.
  • On the CN-E set, the method achieved an Equal Error Rate (EER) of 7.18% and a Minimum Detection Cost Function (MinDCF) of 0.365.
  • The method demonstrated strong performance on the Vox1-O evaluation set, indicating its ability to generalize to new domains not seen during pre-training.
  • Ablation studies showed that the WBDA (Within- and Between-class Distribution Alignment) loss was most effective among tested domain alignment strategies.
  • The optimal number of local prototypes was found to be 128, balancing representation capacity and orthogonality.

Limitations and Future Work

  • Limitations: The paper notes that while the method improves robustness, it still relies on fine-tuning with utterance-level supervision. The impact of BDA (Between-Class Distribution Alignment) was found to be limited, suggesting potential for further refinement in aligning inter-class variations.
  • Future Work: The authors suggest that the learned local prototypes could be further explored for other speech processing tasks. Additionally, investigating more advanced domain adaptation or generalization techniques within the self-supervised framework could lead to even greater robustness.

Summary updated

BibTeX
@inproceedings{gu2025domain,
  author = {Qing Gu and Yan Song and Haoyu Song and Nan Jiang and Lirong Dai and Ian McLoughlin},
  title = {{A Domain Robust Pre-Training Method with Local Prototypes for Speaker Verification}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure