Abstract
We study a novel neural architecture and its training strategies for a speaker encoder for speaker recognition without using any identity labels. The speaker encoder is trained to extract a fixed-size speaker embedding from a spoken utterance of various lengths. Contrastive learning is a typical self-supervised learning technique; however, the quality of the speaker encoder depends heavily on the sampling strategy of positive and negative pairs. It is common to sample a positive pair of segments from the same utterance, known as poor-man's positive pairs (PPP), which lack the diversity necessary for training a robust encoder. In this work, we propose a multi-modal contrastive learning technique with novel sampling strategies. By cross-referencing between speech and face data, we find diverse positive pairs (DPP) for contrastive learning, improving the robustness of the speaker encoder. Our method achieves a state-of-the-art equal error rate (EER) of 1.44% on VoxCeleb1, outperforming existing self-supervised methods and approaching supervised performance.
Introduction
Training robust speaker encoders typically requires massive datasets with manual identity labels. Self-supervised learning (SSL) offers a way to leverage unlabeled data using contrastive learning, where a model learns to pull positive pairs (samples from the same speaker) together and push negative pairs (different speakers) apart.
However, most current SSL methods for speech rely on Poor-man’s Positive Pairs (PPP): two different segments taken from the same utterance. Because these segments share the same background noise, recording device, and room acoustics, the model can "cheat" by focusing on these environmental factors rather than the speaker's unique vocal characteristics. To build a truly robust encoder, the model needs Diverse Positive Pairs (DPP)—samples of the same person speaking in different environments (e.g., an indoor interview vs. an outdoor report). The challenge lies in identifying these diverse pairs without identity labels.
Methods
The proposed framework, Multi-Modal Contrastive Learning with Diverse Positive Pairs (MCL-DPP), addresses the limitations of PPP through two core innovations:
Multi-Modal Cross-Referencing
The system utilizes both audio and video (facial frames). Since a person’s face and voice are intrinsically linked to their identity, the model uses facial similarity to verify that two different audio recordings belong to the same speaker. This visual anchor prevents the audio encoder from making false associations during training.
Progressive Clustering
This curriculum learning strategy starts with a high number of clusters (nearly one per video clip) to ensure high "purity," meaning samples in a cluster are almost certainly the same person. As training progresses and the encoder improves, the number of clusters is gradually reduced. This forces the model to find more "diverse" matches across different video clips that likely belong to the same person, effectively transitioning from PPP to DPP.
The architecture employs an ECAPA-TDNN speaker encoder and a ResNet34 face encoder, with embedding projectors mapping both modalities into a shared space for joint contrastive loss.
Experimental Setup
The model was trained on the VoxCeleb2 dataset (1M+ utterances from ~6k speakers) without identity labels. Robustness was further evaluated on LRS2 and LRW datasets, where speaker identities are unknown. Benchmarking was performed on the VoxCeleb1 (O, E, and H) test sets.
Results
The MCL-DPP method significantly outperforms previous self-supervised benchmarks:
- Single-Stage (MCL-DPP): Achieved an EER of 2.89% on VoxCeleb1-O, a substantial improvement over the previous state-of-the-art (4.83%).
- Two-Stage (MCL-DPP-C): By using the SSL model to generate pseudo-labels for a second round of supervised-style training, the EER dropped to 1.44%.
- Closing the Gap: A fully supervised version of the same encoder achieves 1.01% EER. The MCL-DPP-C framework recovers nearly 90% of the performance of a supervised model without using human-provided labels.
- Generalization: Consistent gains on LRS2 and LRW suggest the sampling strategies are robust across different data distributions.
Conclusion
This work demonstrates that the diversity of positive pairs is more critical than quantity in self-supervised speaker recognition. By using multi-modal data and progressive clustering, we can effectively bridge the gap between self-supervised and supervised learning. Future work will explore alternatives to k-means clustering, such as K-nearest neighbors or similarity thresholds, to further refine the selection of diverse positive pairs and reduce the dependency on video data.
Summary updated