Abstract
Target Speaker Extraction (TSE) aims to isolate a specific speaker's voice from a multi-talker mixture using a short reference enrollment. While previous research has focused heavily on augmenting the mixture signal, this paper explores the impact of augmenting the enrollment speech. We introduce Self-estimated Speech Augmentation (SSA), a technique that utilizes the model's own extracted speech as a reference signal. Our findings demonstrate that enrollment augmentation significantly improves model robustness and performance, particularly in out-of-domain scenarios, yielding gains of up to 2.5 dB in SI-SDR.
Introduction
The "cocktail party problem"—isolating a single voice in a noisy, multi-speaker environment—remains a central challenge in speech processing. Target Speaker Extraction (TSE) addresses this by using a reference "enrollment" clip to identify the target speaker. Traditionally, researchers have sought to improve TSE models by augmenting the input mixture (e.g., adding noise or reverberation). However, this approach often neglects the enrollment signal itself.
If a model is trained only on clean, high-quality enrollment signals, it often fails to generalize when presented with real-world reference audio that may be distorted or noisy. This paper argues that augmenting the enrollment speech is a critical, yet under-utilized, strategy for building robust TSE systems that can handle the variability of practical applications.
Methodology
Enrollment Augmentation Strategies
The study investigates two primary categories of enrollment augmentation:
- Standard Signal Augmentations: Classic techniques including additive noise (MUSAN), simulated room impulse responses (FRAM-RIR), and SpecAugment (time and frequency masking). These methods aim to make the speaker encoder invariant to environmental distortions.
- Self-estimated Speech Augmentation (SSA): This novel approach uses the model's own output—the estimated target speech—as the enrollment signal during training. Because extracted speech often contains specific artifacts and residual interference from other speakers, training the model to recognize the speaker from these "imperfect" signals forces the speaker encoder to focus on robust identity features rather than signal purity.
Model Architecture
The researchers employed a state-of-the-art architecture consisting of a ResNet34 speaker encoder and a Band-Split RNN (BSRNN) extraction module. A key finding during development was that jointly training the speaker encoder with the extraction module outperformed using a fixed, pretrained encoder by approximately 2 dB. This suggests that encoders optimized for general speaker recognition may not produce the specific diversity of embeddings required for the extraction task.
Training and Curriculum Learning
To maintain stability when applying multiple heavy augmentations, the authors utilized a Curriculum Learning (CL) strategy. Training began with clean data, gradually introducing more complex augmentations to ensure the model converged effectively without being overwhelmed by noise early in the process.
Experimental Results
In-Domain Performance
On the Libri2Mix dataset, the proposed SSA method provided the most substantial performance boost, increasing the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) by up to 2.5 dB. The results indicate that enrollment and mixture augmentations are complementary; while mixture augmentation helps the model separate signals, enrollment augmentation ensures the model remains "locked" onto the correct speaker identity.
Cross-Domain Robustness
The effectiveness of enrollment augmentation was most apparent in out-of-domain evaluations. When tested on the Aishell2Mix (Mandarin) dataset—representing a significant shift in language and recording conditions—the baseline model's performance was poor (1.22 dB SI-SDR). However, applying SSA improved this to 4.58 dB, demonstrating a significant leap in generalization capability.
Discussion and Limitations
While most augmentations proved beneficial, SpecAugment yielded mixed results. Although it improved speaker classification accuracy, it occasionally degraded SI-SDR performance, suggesting a potential incompatibility with certain signal-based extraction objectives. Additionally, the complexity of optimizing multiple augmentation types simultaneously requires careful tuning of loss weights to prevent training divergence.
Conclusion
This research highlights that the enrollment signal is just as vital as the mixture signal in the TSE pipeline. By introducing Self-estimated Speech Augmentation (SSA), the authors provide a model-aware method to improve robustness against artifacts and environmental noise. Future work will focus on further integrating these enrollment-side techniques with advanced mixture-side strategies to achieve even greater extraction clarity in diverse acoustic environments.
Summary updated