Discriminative Speaker Representation via Contrastive Learning with Class-Aware Attention in Angular Space

Zhe Li, Man-Wai Mak, Helen Mei-Ling Meng

Introduction

Speaker Verification (SV) systems aim to map audio signals into a high-dimensional embedding space where utterances from the same speaker are clustered together and those from different speakers are distinctly separated. While traditional softmax-based losses are effective for closed-set classification, they often struggle with "open-set" scenarios—verifying speakers not seen during training.

Contrastive learning methods like SimCLR or MoCo have emerged as potential solutions by pulling positive pairs together and pushing negative pairs apart. However, these methods face two significant challenges: a lack of discriminative power due to the absence of a strict decision margin, and sensitivity to "hard negatives" that may actually be mislabeled outliers. This paper introduces CAAMarginCon, a framework designed to address these hurdles through an additive angular margin and a class-aware attention mechanism.

The CAAMarginCon Framework

The proposed Class-Aware Attention-based Contrastive learning with Margin (CAAMarginCon) framework integrates three core components to refine speaker representations:

  1. Supervised Margin Contrastive (SupMarginCon) Loss: The authors extend standard Supervised Contrastive (SupCon) loss by injecting an additive angular margin ($m$). By modifying the cosine similarity during training, the model is forced to push different speakers further apart in angular space, creating a more robust and noise-tolerant decision boundary.
  2. Class-Aware Attention (CAA): To mitigate the "poisoning" effect of outliers, the CAA mechanism compares an embedding to its learned "class vector" (representing the speaker's average voice). If a sample is identified as a noisy outlier, its CAA score decreases, reducing its influence on the total loss. This prevents the model from over-fitting to unrepresentative hard negatives.
  3. Multi-objective Optimization: The framework simultaneously optimizes for contrastive learning and classification using AAMSoftmax (Additive Angular Margin Softmax). A gradient-based technique dynamically balances these objectives, allowing the model to capture both global speaker traits and local utterance relationships.
The CAAMarginCon architecture detailing the training objectives: Angular Margin, Class-Aware Attention (CAA), and AAMSoftmax.

Experimental Results

The model was evaluated on two major benchmarks: CN-Celeb, a challenging "in-the-wild" dataset with significant background noise, and VoxCeleb1, a standard large-scale dataset. To ensure real-world robustness, the researchers employed data augmentation techniques including SpecAugment and environmental noise injection.

As shown in the performance comparison below, CAAMarginCon consistently outperformed baseline models. On the CN-Celeb dataset, the framework achieved an Equal Error Rate (EER) of 8.66%, a significant improvement over standard Cross-Entropy and previous ECAPA-TDNN baselines.

Performance comparison of various networks and loss functions on the CN-Celeb dataset, highlighting CAAMarginCon's state-of-the-art results.

On the VoxCeleb1 dataset, the framework achieved an EER of 2.85%, surpassing the AAMSoftmax baseline of 2.96%. Ablation studies further confirmed that both the angular margin and the CAA module are critical; removing either component led to a measurable degradation in verification performance.

Conclusion

CAAMarginCon demonstrates that integrating angular margins into contrastive objectives significantly enhances the discriminative power of speaker embeddings. Furthermore, the Class-Aware Attention mechanism serves as an effective denoising filter, prioritizing representative samples over noisy outliers. This synergy of classification and contrastive learning provides a robust path forward for high-performance, open-set speaker verification in challenging, real-world environments.

Summary updated

BibTeX
@inproceedings{li2023discriminative,
  author = {Zhe Li and Man-Wai Mak and Helen Mei-Ling Meng},
  title = {{Discriminative Speaker Representation via Contrastive Learning with Class-Aware Attention in Angular Space}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096230},
}
Figure