Inference and Denoise: Causal Inference-based Neural Speech Enhancement

Tsun-An Hsieh, Chao-Han Huck Yang, Pin-Yu Chen, Sabato Marco Siniscalchi, Yu Tsao

TL;DR

This paper introduces Causal Inference-based Speech Enhancement (CISE), a novel framework that treats the presence of noise as a causal intervention rather than a simple correlation. By utilizing a noise detector to dynamically route speech frames to specialized, independent enhancement modules, CISE significantly outperforms traditional non-causal deep learning models in both speech quality and intelligibility while requiring a fraction of the computational overhead.

What Problem Does This Paper Address?

Traditional neural network-based speech enhancement (SE) systems excel at filtering out steady background noise, but they struggle with the intermittent, unpredictable noise typical of real-world environments (e.g., a sudden door slam or a brief drop in a microphone connection). Most modern SE models treat the entire audio waveform uniformly, mapping noisy speech directly to clean speech by learning statistical correlations.

Because they do not explicitly model the presence or absence of noise on a frame-by-frame basis, these models often generalize poorly to unseen acoustic environments. The authors address this gap by reframing speech enhancement through the lens of causal inference (CI), specifically leveraging the potential outcome framework to isolate the true causal effect of noise on speech quality.

Methods

The core innovation of CISE is modeling noise as a binary "treatment" or intervention ($i$) within a Causal Graphical Model (CGM), as illustrated below:

Fig. 1: Causal graphical model (CGM) for the training phase (a) and the testing phase (b). The blue nodes x and y are observable (e.g., noisy speech and speech intelligibility scores). Node z, colored white, is not observable as a parameterized latent variable. Node i, colored blue in (a), is only observable during training, and to be inferred by proposed causal model in (b) the testing time.

The architecture consists of two main components:

  1. Noise Detector: Powered by a WavLM CNN encoder and Conformer blocks, this sub-network predicts the probability ($\hat{p}$) of whether a specific audio frame contains noise. During testing, an intervention mask ($\hat{i}$) is sampled from this probability.
  2. Dual Enhancement Modules (EMs): Instead of a single network handling all audio, CISE uses two completely independent, Conformer-based modules ($EM_{i=0}$ for clean/control frames and $EM_{i=1}$ for noisy/treatment frames) without parameter sharing.
Fig. 2: Proposed causal inference-based SE architecture. Causal inference-based SE (CSIE) is based on two independent neural enhancement module (EM) without parameter sharing.

During training, the true intervention label $i$ guides the system, and the model minimizes a multi-task loss function combining $l_1$ regression distance (for speech reconstruction) and cross-entropy loss (for noise detection).

Additionally, the authors introduce a speech-enhancement-specific metric called the Average Treatment Effect ($ATE_M$). This metric quantifies the causal impact of the noise intervention on standard speech evaluation metrics like PESQ or STOI, providing a formal mathematical framework to evaluate the model's causal reasoning.

Datasets

The framework was evaluated using a curated version of the standard VoiceBank–DEMAND dataset. To make it suitable for causal inference, the authors artificially engineered intermittent noise interventions. They remixed the data by combining clean speech frames and noisy speech frames based on a Bernoulli distribution ($p(i=1) = 0.5$). The training set included 10 types of noise across 4 SNR levels (0 to 15 dB), while the test set featured 5 unseen noises across SNR levels ranging from 2.5 to 17.5 dB to test generalization.

Key Insights and Ideas

  • Causality Beats Pure Correlation: Explicitly separating the "clean speech" concept from the "noisy speech" concept via independent expert modules prevents the network from distorting naturally clean frames, a common issue in vanilla SE models.
  • The Power of Causal Regularization: Even when the noise detector was forced to guess randomly ($Random\ \hat{i}$), the dual-module architecture still outperformed a standard single-module network ($Vanilla$). This suggests that structuring a network around a causal graph acts as a powerful regularizer.
  • The Threshold for Causal Benefit: Through their ATE analysis, the authors discovered a strict threshold: a noise detection accuracy ($p$) of greater than 0.9 is required to yield a positive Average Treatment Effect across all speech metrics. If detection accuracy drops below this, the causal framework can actually degrade performance by routing frames to the wrong module.

Key Results

CISE demonstrated substantial improvements over non-causal baselines and complex state-of-the-art models:

  • Speech Quality: CISE achieved a Perceptual Evaluation of Speech Quality (PESQ) score of 3.15, drastically outperforming the baseline Vanilla model (2.44) and approaching the Oracle model score (3.21), which assumes perfect noise knowledge.
  • Robustness: It consistently beat more complex architectures like CMGAN, DEMUCS, and MANNER across all composite metrics (CSIG, CBAK, COVL, STOI).
  • Efficiency: CISE proved to be incredibly lightweight. Compared to its largest variant (CISE-D), the Conformer-based CISE consumed merely 2.64% of the computational time and 4.96% of the GPU memory, making it highly viable for edge devices and real-time applications.
Table 1: Speech qualities and intelligibility of CISE on the curated VoiceBank–DEMAND dataset. DA is the noise detection accuracy.

Limitations and Future Work

The primary limitation of CISE is its heavy reliance on the accuracy of the noise detector. As shown in the paper's ablation studies, if the domain mismatch causes the noise detector to fail, the entire routing system breaks down.

For future work, the authors point toward moving higher up Pearl’s causal hierarchy into counterfactuals (Level 3). While this paper successfully operates at the intervention level (Level 2), true counterfactual speech enhancement—asking "how would this speech sound if a different noise had occurred?"—remains an open and challenging research frontier.

Summary updated

Abstract

This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. Based on the potential outcome framework, the proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE. Specifically, we use the presence of noise as guidance for EM selection during training, and the noise detector selects the enhancement module according to the prediction of the presence of noise for each frame. Moreover, we derived a SE-specific average treatment effect to quantify the causal effect adequately. Experimental evidence demonstrates that CISE outperforms a non-causal mask-based SE approach in the studied settings and has better performance and efficiency than more complex SE models.
BibTeX
@inproceedings{hsieh2022inference,
  author = {Tsun-An Hsieh and Chao-Han Huck Yang and Pin-Yu Chen and Sabato Marco Siniscalchi and Yu Tsao},
  title = {{Inference and Denoise: Causal Inference-based Neural Speech Enhancement}},
  booktitle = {MLSP},
  year = {2022},
}
Figure