Abstract
Automatic Speech Recognition (ASR) in multi-talker environments, often referred to as the "cocktail party problem," remains a significant challenge. This paper proposes TRnet, a target speaker recovery network that enhances robustness by using average x-vector embeddings rather than single, potentially unreliable audio anchors. Furthermore, the authors introduce global training, a joint optimization strategy for both the recovery and recognition components. Experimental results on the WSJ0-2mix dataset demonstrate a 13% relative improvement in Word Error Rate (WER) and a substantial increase in the Source-to-Distortion Ratio (SDR), proving the effectiveness of stable speaker representations and end-to-end optimization.
Introduction
In environments with multiple simultaneous speakers, ASR systems frequently struggle to isolate the target voice. Current "speaker-aware" approaches typically rely on an anchor—a short segment of clean speech from the target speaker—to guide the extraction process. However, this method is sensitive to the quality and duration of the anchor. Short or noisy anchors often lead to poor recovery. Additionally, there is often a mismatch between the output of the recovery network and the requirements of the downstream ASR acoustic model (AM), where audio that sounds clear to humans may still contain artifacts that degrade recognition accuracy.
Methods
The proposed solution addresses these issues through two primary innovations: stable speaker embeddings and a globally optimized training framework.
Average x-vectors for Stability
Instead of relying on a single random anchor during training, the authors compute an "average x-vector" for each speaker using all available utterances in the training set. This creates a high-dimensional, stable "fingerprint" of the speaker's voice. While a single short anchor can still be used during inference, the model’s training on these stable averages allows it to generalize much better to varying anchor lengths and qualities.
TRnet Architecture
The TRnet architecture is designed to process mixed speech and extract the target signal using the provided speaker embedding. It consists of CNN layers to process the mixed spectrum, BLSTM layers to incorporate the average x-vector, and a Phase-Sensitive Mask (PSM) output to reconstruct the target's clean speech.
Global Training
To bridge the gap between speech recovery and recognition, the authors implemented a joint framework. Rather than treating recovery and ASR as independent modules, the entire pipeline is optimized to minimize the final recognition loss (cross-entropy). This ensures that the recovered speech is specifically tailored for optimal performance by the ASR engine.
Results and Discussion
The research highlights several key findings regarding anchor stability and the benefits of joint optimization.
Impact of Anchor Stability
The study confirmed that longer anchors significantly improve recovery. However, by using average embeddings during training, the model becomes less sensitive to anchor length at test time. Interestingly, averaging just five anchors during inference was found to achieve nearly optimal performance, making the system highly practical for real-world applications.
Performance Metrics
As shown in the results table, the TRnet using average x-vectors improved the Source-to-Distortion Ratio (SDR) by 12.5% (from 9.78 dB to 11.0 dB) compared to a baseline trained on short anchors.
Global Training Success
The global TRnet framework achieved a 13% relative improvement in WER over the baseline. Specifically, global training reduced the WER from 13.79% (using a fine-tuned acoustic model) to 13.19%, demonstrating that end-to-end optimization allows the recovery network to "clean" audio in a way that is most beneficial for the ASR component.
Conclusion
The TRnet framework, combined with average x-vector embeddings and global training, provides a robust solution for target speaker extraction in multi-talker scenarios. Future work will focus on extending this framework to handle more than two speakers and more diverse, extreme noise environments.
Summary updated