Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification

Sung Hwan Mun, Min Hyun Han, Dongjune Lee, Jihwan Kim, Nam Soo Kim

Abstract

This paper proposes a two-stage self-supervised speaker representation learning framework. The front-end utilizes a bootstrap equilibrium strategy with uniformity regularization to learn discriminative features without negative sampling. The back-end employs an uncertainty-aware probabilistic speaker embedding, modeling speakers as Gaussian distributions to account for data noise and variability. Experimental results on the VoxCeleb benchmark demonstrate that this approach outperforms conventional contrastive learning methods, achieving state-of-the-art self-supervised performance.

Introduction

Speaker Verification (SV) systems traditionally rely on large-scale labeled datasets to extract unique voiceprints. Self-supervised learning (SSL) has emerged as a powerful alternative, enabling models to learn from vast amounts of unlabeled audio. However, most SSL techniques for SV are based on contrastive learning, which presents significant challenges in pair selection and representation stability.

The Challenge of Contrastive Learning

Standard SSL methods for speaker verification often rely on contrastive learning frameworks. These require carefully curated "positive pairs" (different segments from the same speaker) and "negative pairs" (segments from different speakers).

Standard contrastive learning framework for speaker representation learning.

As shown in Figure 4, a single utterance $x_i$ is split into segments, augmented, and passed through encoders to compute a contrastive loss. A major drawback of this approach is the risk of "class collisions," where segments from the same speaker are accidentally treated as negative pairs, confusing the model. Additionally, these systems typically produce deterministic embeddings, failing to account for the inherent uncertainty in noisy or short audio clips.

Proposed Methodology

The authors introduce a two-stage framework to address these limitations.

1. Front-End: Bootstrap Equilibrium Learning

To eliminate the need for negative pairs, the authors adopt a bootstrap training scheme inspired by BYOL. This setup uses an online network and a target network. The online network learns to predict the target network's output, which is updated as a moving average of the online network.

To prevent "representation collapse"—where the model outputs the same vector regardless of the input—the authors introduce uniformity regularization. By applying a Gaussian potential kernel, the embeddings are forced to distribute uniformly across a unit hypersphere, maximizing the information content of the learned representations.

2. Back-End: Probabilistic Speaker Embedding

Moving beyond deterministic points, the back-end models each speaker as a Gaussian distribution characterized by a mean (μ) and variance (σ²).

  • Mean: Captures the core identity of the speaker.
  • Variance: Represents data uncertainty, such as background noise or varying utterance lengths.

Verification is performed using a Mutual Likelihood Score (MLS), which calculates the probability that two distributions represent the same underlying speaker identity.

Experimental Setup and Results

The framework was evaluated on the VoxCeleb1 and VoxCeleb2 datasets. Training was conducted in a fully self-supervised manner, utilizing MUSAN noise and RIR filters for data augmentation.

Performance Highlights

  • State-of-the-Art Results: The integrated system achieved an Equal Error Rate (EER) of 6.42% and a MinDCF of 0.345 on the VoxCeleb1 test set.
  • Robustness: Unlike contrastive methods, the bootstrap approach remains effective even with smaller batch sizes, as it does not rely on a large pool of negative samples.
  • Uncertainty Modeling: The probabilistic back-end provided a relative improvement of 5% in EER and 16% in MinDCF over standard cosine similarity, demonstrating the value of modeling "known unknowns" in audio signals.

Conclusion

By combining bootstrap equilibrium learning with probabilistic embeddings, this research provides a robust solution for self-supervised speaker verification. The framework effectively mitigates representation collapse and handles data uncertainty, narrowing the gap between self-supervised and fully supervised SV systems. Future work will explore tighter integration of uncertainty estimation directly into the front-end training process.

Summary updated

Abstract

In this paper, we propose self-supervised speaker representation learning strategies, which comprise of a bootstrap equilibrium speaker representation learning in the front-end and an uncertainty-aware probabilistic speaker embedding training in the back-end. In the front-end stage, we learn the speaker representations via the bootstrap training scheme with the uniformity regularization term. In the back-end stage, the probabilistic speaker embeddings are estimated by maximizing the mutual likelihood score between the speech samples belonging to the same speaker, which provide not only speaker representations but also data uncertainty. Experimental results show that the proposed bootstrap equilibrium training strategy can effectively help learn the speaker representations and outperforms the conventional methods based on contrastive learning. Also, we demonstrate that the integrated two-stage framework further improves the speaker verification performance on the VoxCeleb1 test set in terms of EER and MinDCF.
BibTeX
@inproceedings{mun2021bootstrap,
  author = {Sung Hwan Mun and Min Hyun Han and Dongjune Lee and Jihwan Kim and Nam Soo Kim},
  title = {{Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=vnmtYwuPxK},
}
Figure