Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs

Ruijie Tao, Kong Aik Lee, Rohan Kumar Das, Ville Hautamäki, Haizhou Li

TL;DR

This paper introduces a novel self-supervised learning (SSL) framework for speaker recognition that moves beyond "poor-man’s positive pairs"—segments taken from the same recording. By leveraging multi-modal data (voice and face) and a "progressive clustering" strategy, the authors find "diverse positive pairs" across different recordings, achieving a state-of-the-art 1.44% Equal Error Rate (EER) on VoxCeleb1, nearly closing the gap with supervised learning.

What Problem Does This Paper Address?

Training a robust speaker encoder typically requires massive datasets with manual identity labels. Self-supervised learning (SSL) offers a way to train on unlabeled data using contrastive learning, where a model learns to pull "positive pairs" (samples from the same speaker) together and push "negative pairs" (different speakers) apart.

However, most current SSL methods for speech rely on Poor-man’s Positive Pairs (PPP): two different segments taken from the same utterance. Because these segments share the same background noise, recording device, and room acoustics, the model can "cheat" by focusing on these environmental factors rather than the speaker's unique vocal characteristics. To build a truly robust encoder, the model needs Diverse Positive Pairs (DPP)—samples of the same person speaking in different environments (e.g., an indoor interview vs. an outdoor report). The challenge is: how do you find these diverse pairs without identity labels?

Methods

The authors propose Multi-Modal Contrastive Learning with Diverse Positive Pairs (MCL-DPP). The core innovation lies in two areas:

  1. Multi-Modal Cross-Referencing: The system uses both audio and video (facial frames). Since a person’s face and voice are linked to their identity, the model uses facial similarity to help verify that two different audio recordings belong to the same speaker.
  2. Progressive Clustering: This is a curriculum learning strategy. The model starts with a very high number of clusters (nearly one per video clip), ensuring high "purity" (samples in a cluster are definitely the same person). As training progresses and the encoder improves, the number of clusters is gradually reduced. This forces the model to find more "diverse" matches across different video clips that likely belong to the same person.

The architecture consists of a speaker encoder (ECAPA-TDNN), a face encoder (ResNet34), and embedding projectors that map both modalities into a shared space for joint contrastive loss.

Datasets

  • VoxCeleb2: The primary training set (1M+ utterances from ~6k speakers).
  • VoxMini: A 10% subset of VoxCeleb2 created by the authors for ablation studies.
  • LRS2 & LRW: Real-world "in-the-wild" datasets used to test the model's robustness on data where speaker identities are completely unknown.
  • VoxCeleb1 (O, E, and H sets): Used for evaluation and benchmarking against other state-of-the-art models.

Key Insights and Ideas

  • Diversity Trumps Quantity: The "aha moment" is that a model learns more from a few diverse pairs (same person, different day/mic) than from thousands of easy pairs (same person, same recording).
  • Faces as an "Anchor" for Voice: Using the visual modality acts as a powerful regularizer. It prevents the audio encoder from collapsing or making false associations during the clustering phase.
  • The Purity-Diversity Trade-off: There is a sweet spot in clustering. If you have too many clusters, you lack diversity (PPP). If you have too few, you accidentally group different people together (false positives). Progressive clustering naturally navigates this trade-off.

Key Results

The proposed method significantly outperforms previous self-supervised benchmarks:

  • Single-Stage (MCL-DPP): Achieved an EER of 2.89% on VoxCeleb1-O, a massive improvement over the previous SOTA of 4.83%.
  • Two-Stage (MCL-DPP-C): By using the SSL model to generate pseudo-labels for a second round of supervised-style training, the EER dropped to 1.44%.
  • Closing the Gap: For context, a fully supervised version of the same encoder achieves 1.01% EER. The MCL-DPP-C framework recovers nearly 90% of the performance of a supervised model without using a single human-provided label.
  • Generalization: The model showed consistent gains when trained on LRS2 and LRW, proving it isn't just over-fitting to the VoxCeleb distribution.

Limitations and Future Work

  • Multi-Modal Dependency: The current training strategy requires video. While unlabeled video is abundant (YouTube, etc.), the method cannot currently leverage "audio-only" unlabeled data as effectively.
  • False Negatives in Mini-batches: The authors acknowledge that in very large datasets, two clips of the same person might end up in the same mini-batch but be treated as a "negative pair" because they aren't in the same cluster yet.
  • Future Direction: The authors suggest exploring $K$-nearest neighbors or similarity thresholds as alternatives to $k$-means clustering to further refine how diverse positive pairs are selected.

Summary updated

Abstract

We study a novel neural speaker encoder and its training strategies for speaker recognition without using any identity labels. The speaker encoder is trained to extract a fixed dimensional speaker embedding from a spoken utterance of variable length. Contrastive learning is a typical self-supervised learning technique. However, the contrastive learning of the speaker encoder depends very much on the sampling strategy of positive and negative pairs. It is common that we sample a positive pair of segments from the same utterance. Unfortunately, such a strategy, denoted as poor-man's positive pairs (PPP), lacks the necessary diversity. In this work, we propose a multi-modal contrastive learning technique with novel sampling strategies. By cross-referencing between speech and face data, we find diverse positive pairs (DPP) for contrastive learning, thus improving the robustness of speaker encoder. We train the speaker encoder on the VoxCeleb2 dataset without any speaker labels, and achieve an equal error rate (EER) of 2.89%, 3.17% and 6.27% under the proposed progressive clustering strategy, and an EER of 1.44%, 1.77% and 3.27% under the two-stage learning strategy with pseudo labels, on the three test sets of VoxCeleb1. This novel solution outperforms the state-of-the-art self-supervised learning methods by a large margin, at the same time, achieves comparable results with the supervised learning counterpart. We also evaluate our self-supervised learning technique on the LRS2 and LRW datasets, where speaker information is unavailable. All experiments suggest that the proposed neural architecture and sampling strategies are robust across datasets.
BibTeX
@article{tao2023self,
  author = {Ruijie Tao and Kong Aik Lee and Rohan Kumar Das and Ville Hautamäki and Haizhou Li},
  title = {{Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2023},
  doi = {10.1109/taslp.2023.3268568},
}

Loading related content...

Figure