On the Effectiveness of Enrollment Speech Augmentation For Target Speaker Extraction

Junjie Li, Ke Zhang, Shuai Wang, Haizhou Li, Man-Wai Mak, Kong Aik Lee

Abstract

Target Speaker Extraction (TSE) aims to isolate a specific speaker's voice from a multi-talker mixture using a short reference enrollment. While previous research has focused heavily on augmenting the mixture signal, this paper explores the impact of augmenting the enrollment speech. We introduce Self-estimated Speech Augmentation (SSA), a technique that utilizes the model's own extracted speech as a reference signal. Our findings demonstrate that enrollment augmentation significantly improves model robustness and performance, particularly in out-of-domain scenarios, yielding gains of up to 2.5 dB in SI-SDR.

Introduction

The "cocktail party problem"—isolating a single voice in a noisy, multi-speaker environment—remains a central challenge in speech processing. Target Speaker Extraction (TSE) addresses this by using a reference "enrollment" clip to identify the target speaker. Traditionally, researchers have sought to improve TSE models by augmenting the input mixture (e.g., adding noise or reverberation). However, this approach often neglects the enrollment signal itself.

If a model is trained only on clean, high-quality enrollment signals, it often fails to generalize when presented with real-world reference audio that may be distorted or noisy. This paper argues that augmenting the enrollment speech is a critical, yet under-utilized, strategy for building robust TSE systems that can handle the variability of practical applications.

Methodology

Enrollment Augmentation Strategies

The study investigates two primary categories of enrollment augmentation:

  1. Standard Signal Augmentations: Classic techniques including additive noise (MUSAN), simulated room impulse responses (FRAM-RIR), and SpecAugment (time and frequency masking). These methods aim to make the speaker encoder invariant to environmental distortions.
  2. Self-estimated Speech Augmentation (SSA): This novel approach uses the model's own output—the estimated target speech—as the enrollment signal during training. Because extracted speech often contains specific artifacts and residual interference from other speakers, training the model to recognize the speaker from these "imperfect" signals forces the speaker encoder to focus on robust identity features rather than signal purity.

Model Architecture

The researchers employed a state-of-the-art architecture consisting of a ResNet34 speaker encoder and a Band-Split RNN (BSRNN) extraction module. A key finding during development was that jointly training the speaker encoder with the extraction module outperformed using a fixed, pretrained encoder by approximately 2 dB. This suggests that encoders optimized for general speaker recognition may not produce the specific diversity of embeddings required for the extraction task.

Training and Curriculum Learning

To maintain stability when applying multiple heavy augmentations, the authors utilized a Curriculum Learning (CL) strategy. Training began with clean data, gradually introducing more complex augmentations to ensure the model converged effectively without being overwhelmed by noise early in the process.

Experimental Results

In-Domain Performance

On the Libri2Mix dataset, the proposed SSA method provided the most substantial performance boost, increasing the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) by up to 2.5 dB. The results indicate that enrollment and mixture augmentations are complementary; while mixture augmentation helps the model separate signals, enrollment augmentation ensures the model remains "locked" onto the correct speaker identity.

Cross-Domain Robustness

The effectiveness of enrollment augmentation was most apparent in out-of-domain evaluations. When tested on the Aishell2Mix (Mandarin) dataset—representing a significant shift in language and recording conditions—the baseline model's performance was poor (1.22 dB SI-SDR). However, applying SSA improved this to 4.58 dB, demonstrating a significant leap in generalization capability.

Discussion and Limitations

While most augmentations proved beneficial, SpecAugment yielded mixed results. Although it improved speaker classification accuracy, it occasionally degraded SI-SDR performance, suggesting a potential incompatibility with certain signal-based extraction objectives. Additionally, the complexity of optimizing multiple augmentation types simultaneously requires careful tuning of loss weights to prevent training divergence.

Conclusion

This research highlights that the enrollment signal is just as vital as the mixture signal in the TSE pipeline. By introducing Self-estimated Speech Augmentation (SSA), the authors provide a model-aware method to improve robustness against artifacts and environmental noise. Future work will focus on further integrating these enrollment-side techniques with advanced mixture-side strategies to achieve even greater extraction clarity in diverse acoustic environments.

Summary updated

BibTeX
@inproceedings{li2024effectiveness,
  author = {Junjie Li and Ke Zhang and Shuai Wang and Haizhou Li and Man-Wai Mak and Kong Aik Lee},
  title = {{On the Effectiveness of Enrollment Speech Augmentation For Target Speaker Extraction}},
  booktitle = {SLT},
  year = {2024},
  url = {https://openreview.net/forum?id=yP0Bq0CKeV},
}
Figure