Introduction
Voice separation, often referred to as the "cocktail party problem," remains a significant challenge in signal processing. While deep learning models have improved performance, they often struggle in real-world conditions characterized by heavy noise and reverberation. Traditional auxiliary inputs like visual cues (lip movements) or simple voice activity detection often fall short due to lack of availability or insufficient phonetic detail. This paper proposes a novel transfer learning approach that leverages features from a pre-trained end-to-end automatic speech recognition (E2EASR) system to provide rich phonetic and linguistic context for voice separation.
Leveraging E2EASR Features
The central hypothesis is that a model aware of the phonetic structure of speech can more effectively isolate a target voice. By recognizing the spectral characteristics of specific phonemes (e.g., the shape of a fricative vs. a vowel), the system can "clean" the audio more accurately, even when noise masks parts of the signal.
The authors utilize a hybrid CTC/attention E2EASR system pre-trained on the 960-hour LibriSpeech corpus. Rather than using the final text output, they extract deep features from the BLSTM encoder. These features are particularly effective because they maintain time alignment with the original audio while capturing long-term linguistic dependencies.
System Architecture and Recursive Inference
The separation backbone is the Time-domain Audio Separation Network (TASNet). To bridge the gap between ASR features and audio features, a domain translation network consisting of 1D convolutional layers is employed to adapt the ASR features before they are concatenated with the audio signal inside TASNet.
A key challenge in voice separation is that the clean target voice is unknown at the start of inference. To address this, the authors implement a recursive inference strategy. In the first pass, the model performs separation using zeroed-out ASR features. The resulting estimate is then fed back into the E2EASR system to extract refined features, which are used in a second pass to produce the final, high-quality output.
Experimental Results
The proposed method was evaluated on challenging benchmarks, including a noisy version of the AVSpeech dataset (augmented with Audio-Set noise) and the MUSDB singing voice dataset.
- Speech Separation: The model achieved a Scale-Invariant Signal-to-Distortion Ratio improvement (SI-SDRi) of 12.5 dB, significantly outperforming the audio-only baseline (9.0 dB) and even surpassing state-of-the-art audio-visual models (10.2 dB).
- Singing Voice Separation: Despite being trained on standard speech, the E2EASR features improved singing voice separation from 8.65 dB to 8.80 dB, demonstrating the robustness of phonetic priors across domains.
- Recursive Effectiveness: The gap between "Oracle" features (from clean audio) and "Estimated" features (via recursive inference) was only 0.2 dB, validating the practical utility of the recursive approach for real-world scenarios.
Conclusion
By incorporating high-level phonetic information from pre-trained ASR models, this research demonstrates a significant leap in voice separation performance. The ability to outperform audio-visual models using only audio data makes this approach highly practical for applications where high-quality synchronized video is unavailable. Future work may involve joint optimization of the ASR and separation components or the inclusion of attention weights to further enrich the conditioning signal.
Summary updated