Abstract
This paper proposes a two-stage self-supervised speaker representation learning framework. The front-end utilizes a bootstrap equilibrium strategy with uniformity regularization to learn discriminative features without negative sampling. The back-end employs an uncertainty-aware probabilistic speaker embedding, modeling speakers as Gaussian distributions to account for data noise and variability. Experimental results on the VoxCeleb benchmark demonstrate that this approach outperforms conventional contrastive learning methods, achieving state-of-the-art self-supervised performance.
Introduction
Speaker Verification (SV) systems traditionally rely on large-scale labeled datasets to extract unique voiceprints. Self-supervised learning (SSL) has emerged as a powerful alternative, enabling models to learn from vast amounts of unlabeled audio. However, most SSL techniques for SV are based on contrastive learning, which presents significant challenges in pair selection and representation stability.
The Challenge of Contrastive Learning
Standard SSL methods for speaker verification often rely on contrastive learning frameworks. These require carefully curated "positive pairs" (different segments from the same speaker) and "negative pairs" (segments from different speakers).
As shown in Figure 4, a single utterance $x_i$ is split into segments, augmented, and passed through encoders to compute a contrastive loss. A major drawback of this approach is the risk of "class collisions," where segments from the same speaker are accidentally treated as negative pairs, confusing the model. Additionally, these systems typically produce deterministic embeddings, failing to account for the inherent uncertainty in noisy or short audio clips.
Proposed Methodology
The authors introduce a two-stage framework to address these limitations.
1. Front-End: Bootstrap Equilibrium Learning
To eliminate the need for negative pairs, the authors adopt a bootstrap training scheme inspired by BYOL. This setup uses an online network and a target network. The online network learns to predict the target network's output, which is updated as a moving average of the online network.
To prevent "representation collapse"—where the model outputs the same vector regardless of the input—the authors introduce uniformity regularization. By applying a Gaussian potential kernel, the embeddings are forced to distribute uniformly across a unit hypersphere, maximizing the information content of the learned representations.
2. Back-End: Probabilistic Speaker Embedding
Moving beyond deterministic points, the back-end models each speaker as a Gaussian distribution characterized by a mean (μ) and variance (σ²).
- Mean: Captures the core identity of the speaker.
- Variance: Represents data uncertainty, such as background noise or varying utterance lengths.
Verification is performed using a Mutual Likelihood Score (MLS), which calculates the probability that two distributions represent the same underlying speaker identity.
Experimental Setup and Results
The framework was evaluated on the VoxCeleb1 and VoxCeleb2 datasets. Training was conducted in a fully self-supervised manner, utilizing MUSAN noise and RIR filters for data augmentation.
Performance Highlights
- State-of-the-Art Results: The integrated system achieved an Equal Error Rate (EER) of 6.42% and a MinDCF of 0.345 on the VoxCeleb1 test set.
- Robustness: Unlike contrastive methods, the bootstrap approach remains effective even with smaller batch sizes, as it does not rely on a large pool of negative samples.
- Uncertainty Modeling: The probabilistic back-end provided a relative improvement of 5% in EER and 16% in MinDCF over standard cosine similarity, demonstrating the value of modeling "known unknowns" in audio signals.
Conclusion
By combining bootstrap equilibrium learning with probabilistic embeddings, this research provides a robust solution for self-supervised speaker verification. The framework effectively mitigates representation collapse and handles data uncertainty, narrowing the gap between self-supervised and fully supervised SV systems. Future work will explore tighter integration of uncertainty estimation directly into the front-end training process.
Summary updated