SFSRNet: Super-resolution for Single-Channel Audio Source Separation

Joel Rixen, Matthias RenzAAAI 2022 · DOI

Abstract

Single-channel audio source separation, often referred to as the "cocktail party problem," remains a significant challenge in signal processing. While modern deep learning architectures like SepFormer have achieved impressive results, they are constrained by the computational complexity of processing long audio sequences, often necessitating downsampling that discards critical high-frequency information. This paper introduces SFSRNet, which integrates a dedicated super-resolution (SR) network into the separation pipeline. By leveraging the original mixture as a blueprint for high-frequency reconstruction, SFSRNet achieves a new state-of-the-art SI-SNRi of 24.0 dB on the WSJ0-2mix benchmark, effectively decoupling processing speed from output resolution.

Introduction

The primary objective in audio source separation is to isolate individual speakers from a single-channel recording where multiple voices overlap.

High-level overview of a speech separation system where a mixture of speakers is processed to estimate individual sources.

Modern state-of-the-art models typically utilize Dual-Path architectures (e.g., DPRNN, SepFormer) to capture both local and global dependencies. However, the high sampling rates of audio signals lead to extremely long sequences, creating a bottleneck in memory and computation. To mitigate this, models often downsample input signals, which inherently acts as a low-pass filter, permanently losing high-frequency details that standard upsampling methods cannot recover.

Methodology

SFSRNet addresses the resolution-speed trade-off by building upon the SepFormer architecture with two key innovations: Multi-Resolution Multi-Loss training and a specialized Super-Resolution (SR) module.

Architectural comparison between SepFormer and SFSRNet, including detailed views of the Super-Resolution and Heuristic modules.

Multi-Resolution Multi-Loss

Unlike traditional models that compute loss only at the final output, SFSRNet calculates intermediate estimations after each Transformer block. These estimations are supervised against ground-truth sources at progressively increasing resolutions. This allows the initial layers to focus on coarse, low-frequency structures while subsequent layers refine the high-frequency details.

Super-Resolution Network

The SR network is a post-processing module operating in the frequency domain. It takes three primary inputs:

  1. Downsampled Estimations: The output from the main separation network.
  2. Full-Resolution Mixture: The original input, serving as a reference for high-frequency energy.
  3. Heuristic Estimate: A calculated prior indicating the likely contribution of each source to the mixture's high-frequency bins.

Using 2D convolutional layers, the SR network learns to map the high-frequency energy present in the original mixture onto the separated sources. This approach treats the mixture as a "blueprint," ensuring that the reconstructed high frequencies are physically consistent with the input.

Experimental Results

The model was evaluated on standard benchmarks including WSJ0-2mix and Libri2Mix (Clean and Noisy) at 8 kHz sampling rates.

Performance Benchmarks

On the WSJ0-2mix dataset, SFSRNet established a new state-of-the-art with an SI-SNRi of 24.0 dB, a significant improvement over the baseline SepFormer's 22.3 dB. Performance on Libri2Mix was similarly robust, reaching 21.7 dB on clean data and 16.4 dB on noisy data.

Efficiency and Ablation

A key finding was the model's ability to improve efficiency. By applying the SR network to a downsampled DPRNN model, the authors achieved a 2x speedup while simultaneously surpassing the accuracy of the original full-resolution DPRNN. Furthermore, ablation studies confirmed the necessity of the "heuristic" input, which contributed a 0.4 dB gain in SI-SNRi by providing better guidance for high-frequency attribution.

Discussion and Key Insights

A fundamental insight of this work is that in audio separation, the high-resolution data is not missing—it is simply mixed. The SR network's role is one of "attribution" rather than "hallucination." Qualitatively, the SR module also acts as an error corrector, removing artifacts introduced during the initial separation phase. By using an SR network, researchers can use much higher downsampling factors in the heavy-lifting part of the model to save time and memory, then use a lightweight SR network to restore quality.

Conclusion and Future Work

SFSRNet demonstrates that super-resolution is a viable and effective strategy for scaling audio separation models. Future research will explore phase reconstruction, as the current model primarily focuses on magnitude spectrograms. Additionally, integrating explicit denoising capabilities into the SR process could further enhance performance in highly non-stationary noise environments.

Summary updated

BibTeX
@inproceedings{rixen2022sfsrnet,
  author = {Joel Rixen and Matthias Renz},
  title = {{SFSRNet: Super-resolution for Single-Channel Audio Source Separation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21372},
}
Figure