Self-Supervised Training of Speaker Encoder with Multi-Modal Diverse Positive Pairs

Ruijie Tao, Kong Aik Lee, Rohan Kumar Das, Ville Hautamäki, Haizhou Li

Abstract

We study a novel neural architecture and its training strategies for a speaker encoder for speaker recognition without using any identity labels. The speaker encoder is trained to extract a fixed-size speaker embedding from a spoken utterance of various lengths. Contrastive learning is a typical self-supervised learning technique; however, the quality of the speaker encoder depends heavily on the sampling strategy of positive and negative pairs. It is common to sample a positive pair of segments from the same utterance, known as poor-man's positive pairs (PPP), which lack the diversity necessary for training a robust encoder. In this work, we propose a multi-modal contrastive learning technique with novel sampling strategies. By cross-referencing between speech and face data, we find diverse positive pairs (DPP) for contrastive learning, improving the robustness of the speaker encoder. Our method achieves a state-of-the-art equal error rate (EER) of 1.44% on VoxCeleb1, outperforming existing self-supervised methods and approaching supervised performance.

Introduction

Training robust speaker encoders typically requires massive datasets with manual identity labels. Self-supervised learning (SSL) offers a way to leverage unlabeled data using contrastive learning, where a model learns to pull positive pairs (samples from the same speaker) together and push negative pairs (different speakers) apart.

However, most current SSL methods for speech rely on Poor-man’s Positive Pairs (PPP): two different segments taken from the same utterance. Because these segments share the same background noise, recording device, and room acoustics, the model can "cheat" by focusing on these environmental factors rather than the speaker's unique vocal characteristics. To build a truly robust encoder, the model needs Diverse Positive Pairs (DPP)—samples of the same person speaking in different environments (e.g., an indoor interview vs. an outdoor report). The challenge lies in identifying these diverse pairs without identity labels.

Methods

The proposed framework, Multi-Modal Contrastive Learning with Diverse Positive Pairs (MCL-DPP), addresses the limitations of PPP through two core innovations:

Multi-Modal Cross-Referencing

The system utilizes both audio and video (facial frames). Since a person’s face and voice are intrinsically linked to their identity, the model uses facial similarity to verify that two different audio recordings belong to the same speaker. This visual anchor prevents the audio encoder from making false associations during training.

Progressive Clustering

This curriculum learning strategy starts with a high number of clusters (nearly one per video clip) to ensure high "purity," meaning samples in a cluster are almost certainly the same person. As training progresses and the encoder improves, the number of clusters is gradually reduced. This forces the model to find more "diverse" matches across different video clips that likely belong to the same person, effectively transitioning from PPP to DPP.

Illustration of Diverse Positive Pairs (DPP) sampled from different utterances of the same speaker across varying environments.

The architecture employs an ECAPA-TDNN speaker encoder and a ResNet34 face encoder, with embedding projectors mapping both modalities into a shared space for joint contrastive loss.

Experimental Setup

The model was trained on the VoxCeleb2 dataset (1M+ utterances from ~6k speakers) without identity labels. Robustness was further evaluated on LRS2 and LRW datasets, where speaker identities are unknown. Benchmarking was performed on the VoxCeleb1 (O, E, and H) test sets.

Results

The MCL-DPP method significantly outperforms previous self-supervised benchmarks:

  • Single-Stage (MCL-DPP): Achieved an EER of 2.89% on VoxCeleb1-O, a substantial improvement over the previous state-of-the-art (4.83%).
  • Two-Stage (MCL-DPP-C): By using the SSL model to generate pseudo-labels for a second round of supervised-style training, the EER dropped to 1.44%.
  • Closing the Gap: A fully supervised version of the same encoder achieves 1.01% EER. The MCL-DPP-C framework recovers nearly 90% of the performance of a supervised model without using human-provided labels.
  • Generalization: Consistent gains on LRS2 and LRW suggest the sampling strategies are robust across different data distributions.

Conclusion

This work demonstrates that the diversity of positive pairs is more critical than quantity in self-supervised speaker recognition. By using multi-modal data and progressive clustering, we can effectively bridge the gap between self-supervised and supervised learning. Future work will explore alternatives to k-means clustering, such as K-nearest neighbors or similarity thresholds, to further refine the selection of diverse positive pairs and reduce the dependency on video data.

Summary updated

Abstract

We study a novel neural architecture and its training strategies of speaker encoder for speaker recognition without using any identity labels. The speaker encoder is trained to extract a fixed-size speaker embedding from a spoken utterance of various length. Contrastive learning is a typical self-supervised learning technique. However, the quality of the speaker encoder depends very much on the sampling strategy of positive and negative pairs. It is common that we sample a positive pair of segments from the same utterance. Unfortunately, such poor-man's positive pairs (PPP) lack necessary diversity for the training of a robust encoder. In this work, we propose a multi-modal contrastive learning technique with novel sampling strategies. By cross-referencing between speech and face data, we study a method that finds diverse positive pairs (DPP) for contrastive learning, thus improving the robustness of the speaker encoder. We train the speaker encoder on the VoxCeleb2 dataset without any speaker labels, and achieve an equal error rate (EER) of 2.89\%, 3.17\% and 6.27\% under the proposed progressive clustering strategy, and an EER of 1.44\%, 1.77\% and 3.27\% under the two-stage learning strategy with pseudo labels, on the three test sets of VoxCeleb1. This novel solution outperforms the state-of-the-art self-supervised learning methods by a large margin, at the same time, achieves comparable results with the supervised learning counterpart. We also evaluate our self-supervised learning technique on LRS2 and LRW datasets, where the speaker information is unknown. All experiments suggest that the proposed neural architecture and sampling strategies are robust across datasets.
BibTeX
@article{tao2023self,
  author = {Ruijie Tao and Kong Aik Lee and Rohan Kumar Das and Ville Hautamäki and Haizhou Li},
  title = {{Self-Supervised Training of Speaker Encoder with Multi-Modal Diverse Positive Pairs}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2023},
  doi = {10.1109/taslp.2023.3268568},
}
Figure