Asymmetric Clean Segments-Guided Self-Supervised Learning for Robust Speaker Verification

Chong-Xin Gan, Man‐Wai Mak, Weiwei Lin, Jen‐Tzung Chien

Abstract

Contrastive self-supervised learning (CSL) for speaker verification (SV) has drawn increasing interest recently due to its ability to exploit unlabeled data. Performing data augmentation on raw waveforms, such as adding noise or reverberation, plays a pivotal role in achieving promising results in SV. Data augmentation, however, demands meticulous calibration to ensure intact speaker-specific information, which is difficult to achieve without speaker labels. To address this issue, we introduce a novel framework by incorporating clean and augmented segments into the contrastive training pipeline. The clean segments are repurposed to pair with noisy segments to form additional positive and negative pairs. Moreover, the contrastive loss is weighted to increase the difference between the clean and augmented embeddings of different speakers. Experimental results on Voxceleb1 suggest that the proposed framework can achieve a remarkable 19% improvement over the conventional methods, and it surpasses many existing state-of-the-art techniques.

Introduction

Speaker Verification (SV) systems traditionally rely on massive amounts of human-labeled data. To bypass this dependency, researchers have turned to Self-Supervised Learning (SSL), specifically contrastive learning. In this paradigm, models learn by ensuring that different clips from the same recording (positive pairs) have similar mathematical signatures (embeddings), while clips from different recordings (negative pairs) are pushed apart.

However, two major hurdles persist in standard contrastive learning for SV. First, Augmentation Overkill: while adding noise or reverberation is necessary for robustness, excessive augmentation can wash away unique speaker-specific information, confusing the model. Second, the "Easy Negative" Trap: most negative pairs are easily distinguishable. Without focusing on "hard negatives" (different people with similar voices), the model fails to learn fine-grained boundaries between speaker identities.

Methodology

To address these challenges, this paper proposes the Asymmetric Clean Segments-Guided (ACSG) framework. This approach introduces two primary innovations to the standard contrastive pipeline:

The Triple-Segment Strategy

Instead of comparing only two noisy versions of an utterance, the framework introduces a third, clean (original) segment. This clean segment acts as a grounding anchor. The model is tasked with matching noisy segments not just to each other, but also back to the original clean audio. This ensures that speaker-specific features remain central to the embedding, preventing them from being obscured by heavy data augmentation.

Similarity-Dependent Weighting (W-ACSG)

Recognizing that not all negative pairs provide equal educational value, the authors implemented a weighting mechanism (W-ACSG). This mechanism calculates the similarity between embeddings; if two different speakers sound very similar (a hard negative), the loss function assigns that pair a higher weight. This forces the network to prioritize "pushing" these confusing speakers apart in the embedding space, creating a larger margin between different identities.

The proposed W-ACSG framework featuring asymmetric clean-segment guidance and similarity-dependent weighting.

Experimental Setup

The model was trained on VoxCeleb2, utilizing over 1 million utterances from nearly 6,000 speakers as unlabeled data. Performance was evaluated on the VoxCeleb1 original test set. To simulate real-world conditions, the authors used the MUSAN dataset for ambient noise and Room Impulse Response (RIR) filters to simulate echoes and reverberation.

Key Results

The proposed W-ACSG method demonstrated superior performance compared to several state-of-the-art self-supervised techniques. The system achieved an Equal Error Rate (EER) of 7.02%, a significant improvement over the 8.67% baseline (approximately a 19% relative improvement).

Performance comparison on VoxCeleb1 test set against state-of-the-art self-supervised methods.

Ablation studies confirmed the necessity of both core components. Removing the clean segments caused the EER to rise to 7.96%, while removing the weighting mechanism resulted in an EER of 8.04%. Furthermore, the method proved efficient, achieving these results without the massive batch sizes often required by other high-performing contrastive models.

Discussion and Limitations

While the ACSG framework provides a robust solution for label-free speaker verification, it introduces some computational overhead due to the additional forward passes required for the clean segment. Additionally, this study focused on a single clean segment per utterance; future work could explore if multiple clean segments further enhance performance. Finally, while successful on the VoxCeleb dataset, further research is needed to determine how this clean-guided approach handles extreme domain shifts, such as moving from studio-quality speech to low-bandwidth telephone audio.

Summary updated

Abstract

Contrastive self-supervised learning (CSL) for speaker verification (SV) has drawn increasing interest recently due to its ability to exploit unlabeled data. Performing data augmentation on raw waveforms, such as adding noise or reverberation, plays a pivotal role in achieving promising results in SV. Data augmentation, however, demands meticulous calibration to ensure intact speaker-specific information, which is difficult to achieve without speaker labels. To address this issue, we introduce a novel framework by incorporating clean and augmented segments into the contrastive training pipeline. The clean segments are repurposed to pair with noisy segments to form additional positive and negative pairs. Moreover, the contrastive loss is weighted to increase the difference between the clean and augmented embeddings of different speakers. Experimental results on Voxceleb1 suggest that the proposed framework can achieve a remarkable 19% improvement over the conventional methods, and it surpasses many existing state-of-the-art techniques.
BibTeX
@misc{gan2023asymmetric,
  author = {Chong-Xin Gan and Man‐Wai Mak and Weiwei Lin and Jen‐Tzung Chien},
  title = {{Asymmetric Clean Segments-Guided Self-Supervised Learning for Robust Speaker Verification}},
  year = {2023},
  eprint = {2309.04265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.04265},
}
Figure