A Refining Underlying Information Framework for Monaural Speech Enhancement

Rui Cao, Tianrui Wang, Meng Ge, Longbiao Wang, Jianwu DangarXiv 2023 · DOI · arXiv

Abstract

Modern monaural speech enhancement often relies on "direct-fitting" neural networks that map noisy inputs to clean targets. However, these models frequently suffer from over-suppression or residual noise because they fail to explicitly account for the intrinsic acoustic characteristics of human speech. This paper introduces the Refining Underlying Information (RUI) framework, which treats speech enhancement as an iterative information-gathering process based on the Information Bottleneck (IB) principle. By capturing underlying articulatory attributes and harmonic structures, RUI achieves a significant 0.3+ boost in PESQ scores with minimal computational overhead.

Introduction

Most contemporary speech enhancement systems utilize deep neural networks to perform direct mapping from a noisy spectrum to a clean target. While effective, these models often struggle in complex acoustic environments, leading to the loss of natural speech components (over-suppression) or the retention of unwanted artifacts (under-suppression). The core issue is that these models do not explicitly model the intrinsic acoustic characteristics—such as articulatory structures and harmonics—that make speech sound natural to human ears. The RUI framework addresses this by theoretically and practically incorporating these "hidden" speech properties to repair degraded audio.

Methodology

The RUI framework is designed as an incremental refinement process grounded in the Information Bottleneck (IB) principle. It moves beyond simple spectral mapping to target the "missing" underlying information in speech signals.

Framework Architecture

The framework consists of three primary components:

  1. Pre-enhancement Module (PEM): A baseline model (e.g., DPCRN or CRN) that generates an initial, rough estimate of the clean speech.
  2. Underlying Information Extractor (UIE): Utilizing a "Harmonic Attention" mechanism, this module captures the comb-like harmonic structures inherent in human speech that the PEM might overlook.
  3. Multiple Refinement Iterator (MRI): The core innovation of the framework. Based on the chain rule of entropy, the MRI employs a dual-path residual mechanism—an S-path for spectral features and an A-path for acoustic features—to iteratively "fill in the gaps" of the underlying information.
The RUI framework architecture featuring the Pre-enhancement Module (PEM), Underlying Information Extractor (UIE), and Multiple Refinement Iterator (MRI).

To ensure psychoacoustic fidelity, the framework is optimized using a composite loss function ($L_{AC}$) that combines SI-SNR for signal fidelity and PMSQE for perception-based quality.

Experimental Setup

The framework was rigorously evaluated using the 2020 DNS-Challenge (Deep Noise Suppression) dataset. The training set comprised 500 hours of clean speech and over 180 hours of noise. Final comparisons were conducted on 300 hours of noisy speech with Signal-to-Noise Ratios (SNRs) ranging from -5 dB to 20 dB.

Results and Discussion

The RUI framework demonstrated superior performance across multiple benchmarks:

  • Quality Improvements: RUI consistently improved PESQ scores by over 0.3 compared to baseline models.
  • Efficiency: These gains were achieved with the addition of only 1.18 million parameters. At a total of 2.88M parameters, RUI outperformed significantly larger models like FullSubNet+ (8.67M) and GaGNet (5.94M) in SI-SDR and STOI metrics.
  • Qualitative Analysis: Spectrogram analysis confirms that RUI successfully restores harmonic structures (the distinct horizontal stripes in a spectrogram) that are often blurred or deleted by standard pre-enhancement modules.
Spectrogram comparisons showing how RUI (d) and its iterative refinement steps (e-h) restore harmonic structures compared to the noisy input (a) and the initial PEM output (c).

Conclusion

The RUI framework offers a flexible, "plug-and-play" solution for enhancing existing speech models. By reframing speech enhancement as a compression and refinement task under the Information Bottleneck principle, it effectively captures the underlying acoustic characteristics necessary for natural-sounding speech. Future work will explore more diverse acoustic modeling beyond harmonics to further improve hearing-repair capabilities.

Summary updated

Abstract

Supervised speech enhancement has gained significantly from recent advancements in neural networks, especially due to their ability to non-linearly fit the diverse representations of target speech, such as waveform or spectrum. However, these direct-fitting solutions continue to face challenges with degraded speech and residual noise in hearing evaluations. By bridging the speech enhancement and the Information Bottleneck principle in this letter, we rethink a universal plug-and-play strategy and propose a Refining Underlying Information framework called RUI to rise to the challenges both in theory and practice. Specifically, we first transform the objective of speech enhancement into an incremental convergence problem of mutual information between comprehensive speech characteristics and individual speech characteristics, e.g., spectral and acoustic characteristics. By doing so, compared with the existing direct-fitting solutions, the underlying information stems from the conditional entropy of acoustic characteristic given spectral characteristics. Therefore, we design a dual-path multiple refinement iterator based on the chain rule of entropy to refine this underlying information for further approximating target speech. Experimental results on DNS-Challenge dataset show that our solution consistently improves 0.3+ PESQ score over baselines, with only additional 1.18 M parameters. The source code is available at https://github.com/caoruitju/RUI_SE.
BibTeX
@misc{cao2023refining,
  author = {Rui Cao and Tianrui Wang and Meng Ge and Longbiao Wang and Jianwu Dang},
  title = {{A Refining Underlying Information Framework for Monaural Speech Enhancement}},
  year = {2023},
  eprint = {2312.11201},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.11201},
}
Figure