Speaker Representation Learning via Contrastive Loss with Maximal Speaker Separability

Zhe Li, Man‐Wai Mak

TL;DR

This paper introduces AAMSupCon, a novel supervised contrastive learning framework that enhances speaker representation by maximizing both intra-speaker similarity and inter-speaker separability. By combining the strengths of contrastive learning with an additive angular margin loss, it achieves state-of-the-art performance in speaker verification, particularly for unseen speakers and domains.

What Problem Does This Paper Address?

Learning robust speaker representations is crucial for applications like speaker verification, where the goal is to identify who is speaking. A major challenge is creating models that can accurately distinguish between speakers, even when dealing with new speakers not seen during training or when the audio quality varies (e.g., different acoustic environments, background noise). Traditional methods often struggle with these "unseen" scenarios, leading to reduced generalization performance. The paper aims to address this by developing a learning objective that explicitly encourages a more discriminative embedding space, where representations of the same speaker are tightly clustered, and representations of different speakers are maximally separated.

Methods

The core of this work is the AAMSupCon (Additive Angular Margin Supervised Contrastive) learning framework. It builds upon supervised contrastive learning, which leverages labeled data to pull together embeddings from the same speaker while pushing apart embeddings from different speakers.

Here's a breakdown of the key components:

  • Data Augmentation: For each input audio sample (anchor), the system generates an augmented version. This creates a "positive pair" where both samples are from the same speaker.
  • Encoder Network: An ECAPA-TDNN architecture is used to extract low-dimensional embedding vectors from the audio.
  • Projection Network: A shallow network maps these embeddings into a space where the contrastive loss is applied.
  • Supervised Contrastive Loss (SupCon): This loss function encourages embeddings of the same speaker (positive pairs) to be close and embeddings of different speakers (negative pairs) to be far apart. A key aspect here is that AAMSupCon allows for multiple positive samples per anchor, drawing from other augmented samples of the same speaker within the batch, which is an extension from typical self-supervised contrastive learning.
  • Additive Angular Margin Loss (ArcFace): This component is integrated to further enhance speaker separability. It works by adding an angular margin to the angle between an embedding and its corresponding class (speaker) representation. This forces a larger separation between different speakers, making the decision boundaries more stringent and robust to feature perturbations.

The AAMSupCon loss function combines these elements, aiming to maximize agreement between augmented views of the same speaker while simultaneously pushing apart representations of different speakers with an added angular margin.

Datasets

  • CN-Celeb: This dataset, comprising CN-Celeb1 and CN-Celeb2, was used for both training and evaluation. It's a multi-genre dataset with a large number of speakers and recordings, making it suitable for evaluating speaker recognition systems. The paper utilized over 2,793 speakers from 11 genres for training.

Key Insights and Ideas

  • Leveraging Labels for Contrastive Learning: The paper demonstrates that explicitly using speaker labels within a contrastive learning framework (supervised contrastive learning) is highly effective for speaker representation. This goes beyond typical self-supervised contrastive learning where positive pairs are solely derived from augmentations of the same instance.
  • The Power of Angular Margins: Incorporating an additive angular margin (inspired by ArcFace) into the contrastive loss significantly boosts speaker separability. This geometric approach on a hypersphere helps create more distinct boundaries between speaker embeddings.
  • Multiple Positives per Anchor: Allowing multiple positive samples per anchor in the contrastive loss, drawn from other speakers of the same class in the batch, leads to more tightly clustered speaker representations.
  • Contrastive Power with More Negatives: The denominator of the contrastive loss includes a sum over negative samples. This means that increasing the number of negative samples (other speakers in the batch) naturally enhances the model's ability to distinguish between speakers.

Key Results

The proposed AAMSupCon method achieved state-of-the-art performance on the CN-Celeb evaluation set.

  • EER (Equal Error Rate): AAMSupCon achieved an EER of 8.49%, outperforming the second-best method (AAMSoftmax with ECAPA-TDNN) by 0.30%. This indicates a significant improvement in speaker verification accuracy.
  • minDCF (minimum Detection Cost Function): The method also achieved a minDCF of 0.50, matching the performance of AAMSoftmax and outperforming other baselines.
  • Ablation Study: The ablation study confirmed that AAMSupCon's superior performance is due to the synergistic combination of contrastive learning and the additive angular margin, outperforming individual components like Softmax, AM-Softmax, and even a supervised contrastive loss without the margin.
  • Batch Size Impact: The experiments showed that larger batch sizes lead to better performance in contrastive learning, as they provide more positive and negative samples for the model to learn from.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations beyond the inherent challenges of speaker verification (e.g., short utterances, noisy conditions). The performance is evaluated on a specific dataset (CN-Celeb), and generalization to other datasets or more extreme noise conditions might require further investigation.
  • Future Work: The authors suggest that their framework is easily implementable and provides PyTorch code, encouraging further research and application. The success of AAMSupCon in improving discriminative features for speaker verification opens avenues for exploring its application in other related tasks. The paper also highlights the importance of batch size in contrastive learning, suggesting that future work could explore more efficient ways to leverage large batches or alternative sampling strategies.

Summary updated

Abstract

A great challenge in speaker representation learning using deep models is to design learning objectives that can enhance the discrimination of unseen speakers under unseen domains. This work proposes a supervised contrastive learning objective to learn a speaker embedding space by effectively leveraging the label information in the training data. In such a space, utterance pairs spoken by the same or similar speakers will stay close, while utterance pairs spoken by different speakers will be far apart. For each training speaker, we perform random data augmentation on their utterances to form positive pairs, and utterances from different speakers form negative pairs. To maximize speaker separability in the embedding space, we incorporate the additive angular-margin loss into the contrastive learning objective. Experimental results on CN-Celeb show that this new learning objective can cause ECAPA-TDNN to find an embedding space that exhibits great speaker discrimination. The contrastive learning objective is easy to implement, and we provide PyTorch code at https://github.com/shanmon110/AAMSupCon.
BibTeX
@inproceedings{li2022speaker,
  author = {Zhe Li and Man‐Wai Mak},
  title = {{Speaker Representation Learning via Contrastive Loss with Maximal Speaker Separability}},
  booktitle = {2022},
  year = {2022},
  doi = {10.23919/apsipaasc55919.2022.9980014},
}
Figure