Inference and Denoise: Causal Inference-based Neural Speech Enhancement

Tsun-An Hsieh, Chao-Han Huck Yang, Pin-Yu Chen, Sabato Marco Siniscalchi, Yu Tsao

Abstract

Traditional neural speech enhancement (SE) models typically learn direct mappings from noisy to clean audio, often struggling with intermittent or sporadic real-world noise. This paper introduces CISE (Causal Inference-based Speech Enhancement), a framework that treats noise as a "causal intervention" rather than a mere statistical correlation. By utilizing a noise detector to switch between specialized enhancement modules, CISE achieves state-of-the-art performance with significantly lower computational requirements than larger, non-causal models.

Introduction

Standard SE models often fail to account for "observational uncertainties"—the presence or absence of noise in specific frames. The authors argue that SE should be framed through Causal Inference (CI), where noise is viewed as an intervention affecting the observed signal. By modeling hidden confounding variables that influence both noise and speech quality, CISE aims for a more robust and interpretable system capable of handling complex acoustic environments.

Methodology

The Causal Graphical Model (CGM)

The CISE framework defines four primary variables: the observed noisy speech ($x$), the enhanced speech quality outcome ($y$), a binary intervention label ($i$) indicating noise presence, and a hidden confounding variable ($z$) that impacts the relationship between the others.

Architecture: The Two-Expert Approach

CISE employs a modular design consisting of three main components:

  1. Noise Detector: Utilizes a WavLM CNN encoder and Conformer blocks to predict the probability ($\hat{i}$) of noise in a given frame.
  2. Enhancement Modules (EMs): Two independent Conformer-based modules. $EM_{i=0}$ is optimized for clean or low-noise frames (the control group), while $EM_{i=1}$ handles noisy frames (the treatment group).
  3. The Switch: During inference, the noise detector determines which EM processes the audio frame.
The CISE training and testing flow, illustrating the noise detector and the dual-expert enhancement modules.

Training and Metrics

The system is trained using a multi-task loss function that combines $L_1$ distance for speech reconstruction and Cross-Entropy for noise detection. To quantify the impact of noise on speech quality, the authors derive an Average Treatment Effect (ATE) metric specifically for speech enhancement.

Experimental Setup

The model was evaluated using the VoiceBank–DEMAND dataset. To facilitate causal training, the researchers performed a "causal curation" step, remixing clean and noisy frames using a Bernoulli distribution ($p=0.5$) to create ground-truth intervention labels for every frame.

Results and Discussion

Performance Metrics

CISE demonstrated superior performance across multiple quality and intelligibility metrics. As shown in the table below, CISE achieved a PESQ score of 3.15, significantly outperforming the Vanilla non-causal baseline (2.44) and other complex architectures like CMGAN and DEMUCS.

Comparison of CISE against SOTA models and baselines across various speech quality metrics.

Efficiency and Specialization

A key insight of the CISE framework is that specialization leads to efficiency. By using specialized experts, the model avoids "over-processing" clean speech. Remarkably, CISE requires only 2.64% of the computational time and 4.96% of the GPU memory compared to its largest state-of-the-art competitors while maintaining or exceeding their performance levels.

Conclusion

CISE represents a significant shift in speech enhancement by applying causal inference to handle noise interventions. While the current model uses a binary intervention, future work may explore multi-valued interventions to account for varying noise intensities. The system's high efficiency and performance highlight the potential of specialized, causally-aware neural architectures in real-time audio processing.

Summary updated

Abstract

This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. Based on the potential outcome framework, the proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE. Specifically, we use the presence of noise as guidance for EM selection during training, and the noise detector selects the enhancement module according to the prediction of the presence of noise for each frame. Moreover, we derived an SE-specific average treatment effect to quantify the causal effect adequately. Experimental evidence demonstrates that CISE outperforms a non-causal mask-based SE approach in the studied settings and has better performance and efficiency than more complex SE models. Please find our implementation on GitHub.
BibTeX
@inproceedings{hsieh2022inference,
  author = {Tsun-An Hsieh and Chao-Han Huck Yang and Pin-Yu Chen and Sabato Marco Siniscalchi and Yu Tsao},
  title = {{Inference and Denoise: Causal Inference-based Neural Speech Enhancement}},
  booktitle = {MLSP},
  year = {2022},
}
Figure