Improving Voice Separation by Incorporating End-To-End Speech Recognition

Naoya Takahashi, Mayank Kumar Singh, Sakya Basak, Sudarsanam Parthasaarathy, Sriram Ganapathy, Yuki Mitsufuji

TL;DR

This paper introduces a transfer learning approach that uses features from a pre-trained end-to-end automatic speech recognition (E2EASR) system to improve voice separation. By incorporating long-term phonetic and linguistic context, the proposed method significantly outperforms standard audio-only models and even surpasses audio-visual models that rely on lip-reading data.

What Problem Does This Paper Address?

Voice separation—the "cocktail party problem"—involves isolating a specific voice from a mixture of other speakers or background noise. While deep learning has made massive strides in this field, performance often craters in "in-the-wild" scenarios characterized by heavy noise, reverberation, or limited training data.

Previous attempts to bolster separation have used auxiliary information like visual cues (lip movements) or voice activity detection. However, cameras aren't always available, and simple "is someone talking?" signals lack the rich phonetic detail needed to reconstruct a voice accurately. The authors argue that if a model "knows" the phonetic structure of what is being said (e.g., recognizing the spectral shape of a fricative vs. a vowel), it can more effectively "clean" the target voice from a noisy mixture.

Methods

The core innovation is the use of E2EASR features as a conditioning signal for a separation network.

  1. Transfer Learning: The authors use an E2EASR system (based on a hybrid CTC/attention architecture) pre-trained on a massive speech corpus.
  2. Feature Extraction: Instead of using the final text output, they extract deep features from the ASR's BLSTM encoder. These features are ideal because they maintain time alignment with the audio while encoding long-term linguistic dependencies.
  3. The Separation Network: They employ TASNet (Time-domain Audio Separation Network), a state-of-the-art model for speech separation.
  4. Domain Translation: Because ASR features and audio features live in different mathematical spaces, a small "domain translation network" (1D convolutional layers) is used to adapt the ASR features before they are concatenated with the audio signal inside TASNet.
  5. Recursive Inference: Since the target voice is unknown at the start, the model performs a "first pass" separation (using zeroed-out ASR features). It then extracts ASR features from this initial estimate and runs the separation again to produce the final, refined output.

Datasets

  • ASR Pre-training: The LibriSpeech corpus (960 hours of clean speech).
  • Speech Separation & Enhancement: A subset of AVSpeech (100 hours for training, 15 hours for testing). To make it realistic, they added noise from Audio-Set, creating a much more challenging benchmark than standard "clean" laboratory datasets.
  • Singing Voice Separation (SVS): The MUSDB dataset, which contains only about 6.7 hours of training data—a "low-resource" scenario designed to test the robustness of the transfer learning.

Key Insights and Ideas

  • Phonetic Priors as a "Safety Net": By understanding the phonetic nature of the speech, the model can leverage prior knowledge of spectrogram shapes. If the noise is so loud that a specific sound is masked, the ASR features provide a "hint" of what should be there.
  • Long-term Context Matters: Traditional ASR models often look at just a few frames. E2EASR models look at much longer sequences, allowing the separation model to resolve ambiguities by understanding the broader linguistic context.
  • Cross-Domain Robustness: One of the most surprising findings was that ASR features trained on standard speech improved the separation of singing voices. Despite the differences in pitch, timbre, and tempo, the underlying phonetic information remained useful.
  • Audio-Only Beats Audio-Visual: The proposed method outperformed models using visual (lip-reading) features. This is a major win for practicality, as high-quality synchronized video is much harder to obtain than audio.

Key Results

  • Speech Separation: The proposed method achieved a Scale-Invariant Signal-to-Distortion Ratio improvement (SI-SDRi) of 12.5 dB.
  • This significantly beat the audio-only baseline (9.0 dB).
  • It also outperformed the audio-visual model (10.2 dB).
  • Singing Voice Separation: Even with the massive domain mismatch, the E2EASR features improved the SDR from 8.65 dB (baseline) to 8.80 dB.
  • Practicality: The "Oracle" version (using features from the perfect, clean target) scored 12.7 dB, while the "Estimated" version (using the recursive pass) scored 12.5 dB. This tiny 0.2 dB gap proves the recursive inference strategy is highly effective for real-world use where the clean target isn't known.

Limitations and Future Work

  • Computational Overhead: The recursive inference process requires running parts of the model twice, which increases latency—a potential hurdle for real-time applications.
  • Feature Selection: The authors noted that they only used encoder outputs. They suggest that attention weights from the ASR system might provide even richer conditioning information for future models.
  • Joint Optimization: In this study, the ASR model was kept frozen. The authors suggest that jointly training the ASR and the separation network could lead to even tighter integration and better performance.

Summary updated

Abstract

Despite recent advances in voice separation methods, many challenges remain in realistic scenarios such as noisy recording and the limits of available data. In this work, we propose to explicitly incorporate the phonetic and linguistic nature of speech by taking a transfer learning approach using an end-to-end automatic speech recognition (E2EASR) system. The voice separation is conditioned on deep features extracted from E2EASR to cover the long-term dependence of phonetic aspects. Experimental results on speech separation and enhancement task on the AVSpeech dataset show that the proposed method significantly improves the signal-to-distortion ratio over the baseline model and even outperforms an audio visual model, that utilizes visual information of lip movements.
BibTeX
@inproceedings{takahashi2020improving,
  author = {Naoya Takahashi and Mayank Kumar Singh and Sakya Basak and Sudarsanam Parthasaarathy and Sriram Ganapathy and Yuki Mitsufuji},
  title = {{Improving Voice Separation by Incorporating End-To-End Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053845},
}
Figure