Target Speaker Recovery and Recognition Network with Average x-Vector and Global Training

Wenjie Li, Pengyuan Zhang, Yonghong Yan

Abstract

Automatic Speech Recognition (ASR) in multi-talker environments, often referred to as the "cocktail party problem," remains a significant challenge. This paper proposes TRnet, a target speaker recovery network that enhances robustness by using average x-vector embeddings rather than single, potentially unreliable audio anchors. Furthermore, the authors introduce global training, a joint optimization strategy for both the recovery and recognition components. Experimental results on the WSJ0-2mix dataset demonstrate a 13% relative improvement in Word Error Rate (WER) and a substantial increase in the Source-to-Distortion Ratio (SDR), proving the effectiveness of stable speaker representations and end-to-end optimization.

Introduction

In environments with multiple simultaneous speakers, ASR systems frequently struggle to isolate the target voice. Current "speaker-aware" approaches typically rely on an anchor—a short segment of clean speech from the target speaker—to guide the extraction process. However, this method is sensitive to the quality and duration of the anchor. Short or noisy anchors often lead to poor recovery. Additionally, there is often a mismatch between the output of the recovery network and the requirements of the downstream ASR acoustic model (AM), where audio that sounds clear to humans may still contain artifacts that degrade recognition accuracy.

Methods

The proposed solution addresses these issues through two primary innovations: stable speaker embeddings and a globally optimized training framework.

Average x-vectors for Stability

Instead of relying on a single random anchor during training, the authors compute an "average x-vector" for each speaker using all available utterances in the training set. This creates a high-dimensional, stable "fingerprint" of the speaker's voice. While a single short anchor can still be used during inference, the model’s training on these stable averages allows it to generalize much better to varying anchor lengths and qualities.

TRnet Architecture

The TRnet architecture is designed to process mixed speech and extract the target signal using the provided speaker embedding. It consists of CNN layers to process the mixed spectrum, BLSTM layers to incorporate the average x-vector, and a Phase-Sensitive Mask (PSM) output to reconstruct the target's clean speech.

The TRnet architecture showing the integration of average x-vectors and the global training loop.

Global Training

To bridge the gap between speech recovery and recognition, the authors implemented a joint framework. Rather than treating recovery and ASR as independent modules, the entire pipeline is optimized to minimize the final recognition loss (cross-entropy). This ensures that the recovered speech is specifically tailored for optimal performance by the ASR engine.

Results and Discussion

The research highlights several key findings regarding anchor stability and the benefits of joint optimization.

Impact of Anchor Stability

The study confirmed that longer anchors significantly improve recovery. However, by using average embeddings during training, the model becomes less sensitive to anchor length at test time. Interestingly, averaging just five anchors during inference was found to achieve nearly optimal performance, making the system highly practical for real-world applications.

Performance Metrics

As shown in the results table, the TRnet using average x-vectors improved the Source-to-Distortion Ratio (SDR) by 12.5% (from 9.78 dB to 11.0 dB) compared to a baseline trained on short anchors.

Performance comparison of different TRnet configurations in terms of WER and SDR.

Global Training Success

The global TRnet framework achieved a 13% relative improvement in WER over the baseline. Specifically, global training reduced the WER from 13.79% (using a fine-tuned acoustic model) to 13.19%, demonstrating that end-to-end optimization allows the recovery network to "clean" audio in a way that is most beneficial for the ASR component.

Conclusion

The TRnet framework, combined with average x-vector embeddings and global training, provides a robust solution for target speaker extraction in multi-talker scenarios. Future work will focus on extending this framework to handle more than two speakers and more diverse, extreme noise environments.

Summary updated

Abstract

It is very challenging to do multi-talker automatic speech recognition (ASR). Some speaker-aware selective methods have been proposed to recover the speech of the target speaker, relying on the auxiliary speaker information provided by an anchor (a clean audio sample of the target speaker). But the performance is unstable depending on the quality of the provided anchors. To address this limitation, we propose to take advantage of the average speaker embeddings to build the target speaker recovery network (TRnet). The TRnet takes the mixed speech and the stable average speaker embeddings to produce the TF masks for the target speech. During training of the TRnet, we summarize the speaker embeddings on the whole training dataset for each speaker, instead of extracting on a randomly picked anchor. On the testing stage, one or very few anchors are enough to get decent recovery results. The results of the TRnet trained with average speaker embeddings show 13% and 12.5% relative improvements on WER and SDR, compared with the short-anchor trained model. Moreover, to mitigate the mismatch between the TRnet and the acoustic model (AM), we adopted two strategies: fine-tuning the AM and training an global TRnet. Both of them bring considerable reductions on WER. The results show that the global trained framework gets superior performance.
BibTeX
@inproceedings{li2019target,
  author = {Wenjie Li and Pengyuan Zhang and Yonghong Yan},
  title = {{Target Speaker Recovery and Recognition Network with Average x-Vector and Global Training}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1692},
}
Figure