Abstract
Speech Enhancement (SE) systems typically operate on monaural input and are used for applications including voice communications and capture cleanup for user generated content. Recent advancements and changes in the devices used for these applications are likely to lead to an increase in the amount of two-channel content for the same applications. However, SE systems are typically designed for monaural input; stereo results produced using trivial methods such as channel independent or mid-side processing may be unsatisfactory, including substantial speech distortions. To address this, we propose a system which creates a novel representation of stereo signals called Custom Mid-Side Signals (CMSS). CMSS allow benefits of mid-side signals for center-panned speech to be extended to a much larger class of input signals. This in turn allows any existing monaural SE system to operate as an efficient stereo system by processing the custom mid signal. We describe how the parameters needed for CMSS can be efficiently estimated by a component of the spatio-level filtering source separation system. Subjective listening using state-of-the-art deep learning-based SE systems on stereo content with various speech mixing styles shows that CMSS processing leads to improved speech quality at approximately half the cost of channel-independent processing.
Introduction
Most state-of-the-art speech enhancement (SE) systems are designed for monaural (single-channel) audio. As mobile devices and laptops increasingly record in stereo, applying these mono models to stereo signals becomes a challenge. The "naive" approach of channel-independent processing—running the mono model on the Left and Right channels separately—often fails due to inconsistent noise suppression between channels, leading to "noise leakage," spatial instability, and phase distortions. Furthermore, running two instances of heavy deep-learning models is computationally expensive for mobile devices.
Methods
The proposed system utilizes Custom Mid-Side Signals (CMSS) to enable efficient stereo processing with a single monaural model. The process involves four main stages:
- Spatially Identifiable Subband Source Detection (SISSD): The system first analyzes the stereo input to estimate two key parameters: the Interchannel Level Difference ($\Theta_1$) and the Interchannel Phase Difference ($\Phi_1$).
- Custom Mid-Side Generation: CMSS uses these estimated parameters to create a "Custom Mid" (CMS) signal. This signal is mathematically optimized to boost the target speech, regardless of where it is panned in the stereo field, while the "Custom Side" signal suppresses it.
- Monaural Processing: A standard monaural SE model (e.g., U-NetFB or DPCRN) processes only the Custom Mid signal. The Custom Side signal is discarded (set to zero).
- Stereo Reconstruction: Finally, the system uses inversion equations to turn the enhanced mono signal back into a stereo signal, preserving the original spatial positioning of the speaker.
Evaluation and Datasets
The authors evaluated the system using 12 diverse audio scenarios designed to test different mixing styles and environments. These items, categorized as Professionally Generated Content (PGC), include various signal-to-noise ratios (SNR), mixing styles (center-panned, panned left-center, etc.), and genres ranging from sports to scripted movies.
Key Results
Subjective listening tests conducted with nine trained experts yielded decisive results in favor of the CMSS approach:
- Preference: The CMSS system was preferred 73% of the time over the standard channel-independent baseline.
- Speech Quality: Subjects reported better speech quality (less distortion) in 64% of the comparisons.
- Noise Reduction: The system produced fewer perceptible non-speech artifacts in 72% of cases.
- Efficiency: By processing only one channel (the CMS) instead of two (L and R), the system reduces the computational cost of deep learning inference by approximately 50%.
- Spatial Integrity: Despite only processing one channel, there was no significant loss in spatial quality; the speaker remained correctly positioned in the stereo field.
Conclusion and Future Work
CMSS provides a high-quality, efficient alternative for stereo speech enhancement by effectively "steering" the audio field so that speech always appears centered for the monaural model. Future work will address granularity trade-offs in frequency band estimation to improve performance in heavy reverb and explore whether the discarded "Side" signal can be utilized in complex mixing scenarios. Additionally, the study highlights the need for new, spatially-aware automated benchmarks for stereo audio evaluation.
Summary updated