Abstract
Traditional neural speech enhancement (SE) models typically learn direct mappings from noisy to clean audio, often struggling with intermittent or sporadic real-world noise. This paper introduces CISE (Causal Inference-based Speech Enhancement), a framework that treats noise as a "causal intervention" rather than a mere statistical correlation. By utilizing a noise detector to switch between specialized enhancement modules, CISE achieves state-of-the-art performance with significantly lower computational requirements than larger, non-causal models.
Introduction
Standard SE models often fail to account for "observational uncertainties"—the presence or absence of noise in specific frames. The authors argue that SE should be framed through Causal Inference (CI), where noise is viewed as an intervention affecting the observed signal. By modeling hidden confounding variables that influence both noise and speech quality, CISE aims for a more robust and interpretable system capable of handling complex acoustic environments.
Methodology
The Causal Graphical Model (CGM)
The CISE framework defines four primary variables: the observed noisy speech ($x$), the enhanced speech quality outcome ($y$), a binary intervention label ($i$) indicating noise presence, and a hidden confounding variable ($z$) that impacts the relationship between the others.
Architecture: The Two-Expert Approach
CISE employs a modular design consisting of three main components:
- Noise Detector: Utilizes a WavLM CNN encoder and Conformer blocks to predict the probability ($\hat{i}$) of noise in a given frame.
- Enhancement Modules (EMs): Two independent Conformer-based modules. $EM_{i=0}$ is optimized for clean or low-noise frames (the control group), while $EM_{i=1}$ handles noisy frames (the treatment group).
- The Switch: During inference, the noise detector determines which EM processes the audio frame.
Training and Metrics
The system is trained using a multi-task loss function that combines $L_1$ distance for speech reconstruction and Cross-Entropy for noise detection. To quantify the impact of noise on speech quality, the authors derive an Average Treatment Effect (ATE) metric specifically for speech enhancement.
Experimental Setup
The model was evaluated using the VoiceBank–DEMAND dataset. To facilitate causal training, the researchers performed a "causal curation" step, remixing clean and noisy frames using a Bernoulli distribution ($p=0.5$) to create ground-truth intervention labels for every frame.
Results and Discussion
Performance Metrics
CISE demonstrated superior performance across multiple quality and intelligibility metrics. As shown in the table below, CISE achieved a PESQ score of 3.15, significantly outperforming the Vanilla non-causal baseline (2.44) and other complex architectures like CMGAN and DEMUCS.
Efficiency and Specialization
A key insight of the CISE framework is that specialization leads to efficiency. By using specialized experts, the model avoids "over-processing" clean speech. Remarkably, CISE requires only 2.64% of the computational time and 4.96% of the GPU memory compared to its largest state-of-the-art competitors while maintaining or exceeding their performance levels.
Conclusion
CISE represents a significant shift in speech enhancement by applying causal inference to handle noise interventions. While the current model uses a binary intervention, future work may explore multi-valued interventions to account for varying noise intensities. The system's high efficiency and performance highlight the potential of specialized, causally-aware neural architectures in real-time audio processing.
Summary updated