Introduction
Most modern speech enhancement systems use Deep Neural Networks (DNNs) to produce a single "best guess"—usually a multiplicative mask—to clean up noisy audio. However, these models are often overconfident, providing an output even when the input is so noisy that a reliable estimate is impossible. In safety-critical or high-stakes applications like hearing aids or automated transcription, knowing the predictive uncertainty is crucial. This uncertainty is typically split into two types:
- Aleatoric Uncertainty: Inherent randomness in the data (e.g., speech and noise overlapping perfectly).
- Epistemic Uncertainty: Uncertainty in the model itself (e.g., the model hasn't encountered a specific noise type before).
Existing methods for measuring epistemic uncertainty, such as Bayesian neural networks, often require multiple forward passes (sampling), which is too slow for real-time speech processing. This paper introduces a framework to obtain both types of uncertainty instantly in a single forward pass.
Methods
The authors propose a "one-to-many" mapping by modeling the clean speech posterior as a mixture of multiple complex Gaussian components.
Instead of predicting a single mask, the DNN predicts the parameters of a Complex Gaussian Mixture Model (CGMM): the weights, the Wiener filters (means), and the variances for multiple potential "hypotheses" of what the clean speech might be. Using the Law of Total Variance, the total output variance is decomposed: the average variance across components represents aleatoric uncertainty, while the variance between the different component means represents epistemic uncertainty.
To ensure training stability, the authors implemented gradient modifications to prevent numerical instability and Winner-Takes-All (WTA) pre-training. WTA prevents "mode collapse" by initially training the network to only update the component that provides the best prediction for a given sample, forcing components to specialize in different hypotheses.
Key Insights and Results
The models were evaluated using the DNS Challenge Dataset and the WSJ0-CHiME synthetic test set. A key insight is treating mixture components as different "reasonable hypotheses." If the components agree, epistemic uncertainty is low; if they diverge, the model signals confusion.
On the DNS test set, the proposed CGMM4-pre model achieved a PESQ score of 2.69, significantly outperforming the standard Wiener Filter baseline (2.48). Furthermore, sparsification plots demonstrated that these uncertainty estimates are highly correlated with actual reconstruction errors. When the model reports high uncertainty, the error is indeed higher, providing a reliable confidence measure. The mixture-based approach showed particular strength in low SNR (high-noise) scenarios, remaining robust by maintaining multiple hypotheses.
Limitations and Future Work
While WTA pre-training mitigates mode collapse, training mixture models remains more complex than standard regression. Future work could involve integrating these confidence scores into downstream tasks, such as "gating" speech recognition systems or assisting multi-modal systems in deciding when to rely on audio versus visual cues. Additionally, while theoretically efficient, detailed benchmarking of FLOP counts and latency for the additional mixture parameters would be beneficial for real-time deployment.
Summary updated