Abstract
Single-channel audio source separation, often referred to as the "cocktail party problem," remains a significant challenge in signal processing. While modern deep learning architectures like SepFormer have achieved impressive results, they are constrained by the computational complexity of processing long audio sequences, often necessitating downsampling that discards critical high-frequency information. This paper introduces SFSRNet, which integrates a dedicated super-resolution (SR) network into the separation pipeline. By leveraging the original mixture as a blueprint for high-frequency reconstruction, SFSRNet achieves a new state-of-the-art SI-SNRi of 24.0 dB on the WSJ0-2mix benchmark, effectively decoupling processing speed from output resolution.
Introduction
The primary objective in audio source separation is to isolate individual speakers from a single-channel recording where multiple voices overlap.
Modern state-of-the-art models typically utilize Dual-Path architectures (e.g., DPRNN, SepFormer) to capture both local and global dependencies. However, the high sampling rates of audio signals lead to extremely long sequences, creating a bottleneck in memory and computation. To mitigate this, models often downsample input signals, which inherently acts as a low-pass filter, permanently losing high-frequency details that standard upsampling methods cannot recover.
Methodology
SFSRNet addresses the resolution-speed trade-off by building upon the SepFormer architecture with two key innovations: Multi-Resolution Multi-Loss training and a specialized Super-Resolution (SR) module.
Multi-Resolution Multi-Loss
Unlike traditional models that compute loss only at the final output, SFSRNet calculates intermediate estimations after each Transformer block. These estimations are supervised against ground-truth sources at progressively increasing resolutions. This allows the initial layers to focus on coarse, low-frequency structures while subsequent layers refine the high-frequency details.
Super-Resolution Network
The SR network is a post-processing module operating in the frequency domain. It takes three primary inputs:
- Downsampled Estimations: The output from the main separation network.
- Full-Resolution Mixture: The original input, serving as a reference for high-frequency energy.
- Heuristic Estimate: A calculated prior indicating the likely contribution of each source to the mixture's high-frequency bins.
Using 2D convolutional layers, the SR network learns to map the high-frequency energy present in the original mixture onto the separated sources. This approach treats the mixture as a "blueprint," ensuring that the reconstructed high frequencies are physically consistent with the input.
Experimental Results
The model was evaluated on standard benchmarks including WSJ0-2mix and Libri2Mix (Clean and Noisy) at 8 kHz sampling rates.
Performance Benchmarks
On the WSJ0-2mix dataset, SFSRNet established a new state-of-the-art with an SI-SNRi of 24.0 dB, a significant improvement over the baseline SepFormer's 22.3 dB. Performance on Libri2Mix was similarly robust, reaching 21.7 dB on clean data and 16.4 dB on noisy data.
Efficiency and Ablation
A key finding was the model's ability to improve efficiency. By applying the SR network to a downsampled DPRNN model, the authors achieved a 2x speedup while simultaneously surpassing the accuracy of the original full-resolution DPRNN. Furthermore, ablation studies confirmed the necessity of the "heuristic" input, which contributed a 0.4 dB gain in SI-SNRi by providing better guidance for high-frequency attribution.
Discussion and Key Insights
A fundamental insight of this work is that in audio separation, the high-resolution data is not missing—it is simply mixed. The SR network's role is one of "attribution" rather than "hallucination." Qualitatively, the SR module also acts as an error corrector, removing artifacts introduced during the initial separation phase. By using an SR network, researchers can use much higher downsampling factors in the heavy-lifting part of the model to save time and memory, then use a lightweight SR network to restore quality.
Conclusion and Future Work
SFSRNet demonstrates that super-resolution is a viable and effective strategy for scaling audio separation models. Future research will explore phase reconstruction, as the current model primarily focuses on magnitude spectrograms. Additionally, integrating explicit denoising capabilities into the SR process could further enhance performance in highly non-stationary noise environments.
Summary updated