Speech-enhancement networks that report how unsure they are about each cleaned-up time-frequency cell have started to appear. Their uncertainty is almost always assumed independent from one frequency bin to the next. That assumption is harmless when the uncertainty only softens each bin's own filter. It becomes harmful when the uncertainty is passed downstream to a speech recognizer. A recognizer does not read individual bins. It reads mel-band energies, which are weighted sums over many adjacent bins. The variance of a sum depends on the covariances between its terms. Voiced speech couples bins through harmonics, and window leakage couples neighbouring bins, so their errors are positively correlated. A diagonal model therefore systematically understates feature uncertainty exactly in the voiced, speech-dominated frames where the recognizer most needs it.
We propose to have the enhancement front-end predict a posterior over clean speech whose covariance across frequency is low-rank plus diagonal. The low-rank directions are tied to the harmonic comb of the estimated pitch and to banded leakage. That structured covariance is then pushed through the recognizer with an unscented transform whose sigma points (a small, deterministic set of sample inputs chosen to match a distribution's mean and covariance) lie along those few directions. The claim is that, once propagation is done through the whole network, the remaining bottleneck in uncertainty-aware recognition is the structure of the input uncertainty rather than the propagation algorithm. A cross-frequency covariance with a handful of physically motivated factors should close a measurable part of that gap at a cost of $2K+1$ passes, not hundreds.
Why this gap, and what already exists
Uncertainty-aware recognition is an old idea. Propagation of uncertainty through multilayer perceptrons for robust automatic speech recognition pushed Gaussian feature uncertainty through MLPs with the unscented transform and a piecewise-exponential sigmoid approximation. It reported relative word error rate (WER) reductions of 20.8% (Monte Carlo), 17.9% (unscented transform) and 15.5% (piecewise-exponential) on AURORA4. Uncertainty propagation through deep neural networks then showed on CHiME-2 that layer-by-layer propagation under a hidden-unit independence assumption is worse than doing nothing (23.82% WER versus a 21.67% baseline). Treating the whole network as one non-linear map helps: 21.23% with an entire-network unscented transform and 21.06% with Monte Carlo. These papers supply our propagation method: we propagate through the entire acoustic model, never layer by layer. They also motivate the proposal, because the gains on CHiME-2 are small, and in both papers the input uncertainty is diagonal across spectral features. We hypothesise that the diagonal input is one reason the gains stay small.
On the enhancement side, two papers predict a posterior rather than a point estimate. Integrating Uncertainty into Neural Network-based Speech Enhancement trains a Conv-TasNet to output a Wiener-filter mean and a per-bin variance, and uses them in an approximate MAP estimator (POLQA 3.06 versus 2.88–2.93 for point-estimate baselines). Uncertainty Estimation in Deep Speech Enhancement Using Complex Gaussian Mixture Models extends this to a per-bin complex Gaussian mixture that separates aleatoric from epistemic uncertainty (PESQ 2.69 versus 2.48). Both assume independent bins, and both consume their uncertainty inside the enhancer. We keep their training recipe, a negative log-posterior loss plus a signal-level loss. We change what the posterior represents and who consumes it.
The cross-frequency literature supplies the structure we put into the covariance. Meta-Learning for Adaptive Filters with Higher-Order Frequency Dependencies showed that coupling neighbouring bins through banded groupings beats independent-bin filtering by 3.21 dB SI-SDR. Its models are deterministic, and they are limited to local bands. A Convolutional Neural Network with Non-Local Module for Speech Enhancement and Variational Speech Separation of More Sources than Mixtures argue that distant harmonics carry the decisive cues. DPCRN: Dual-Path Convolution Recurrent Network for Single Channel Speech Enhancement runs a recurrent network along the frequency axis of each frame to capture this. We use DPCRN as the backbone. Its frequency-axis path already sees the whole frame, so it is a natural place to read off covariance factors. The non-local and banded papers tell us what shape those factors should have: harmonic combs plus local bands.
The closest bridge, Uncertainty-Driven Spectral Compressive Imaging with Spatial-Frequency Transformer, combines frequency-wise attention with an uncertainty map. It uses that map only to weight the training loss, so nothing is propagated at test time. Input uncertainty propagation through trained neural networks propagates rich input distributions with Wasserstein-guided Gaussian mixtures, but only on image benchmarks and without frequency structure. Neither predicts or propagates correlated spectral uncertainty.
The environmental-modelling half of our first direction shows the same mechanism at a larger scale. Bayesian Analysis of Uncertainty in the GlobCover 2009 Land Cover Product at Climate Model Grid Scale studies how pixel-level misclassification behaves once aggregated to coarse grids. Simulated high-latitude soil thermal dynamics during the past 4 decades finds that forcing-data choice outweighs model physics. Both say that errors in the inputs, and how they combine under aggregation, can dominate the errors of the model itself. Speech recognition fed by an enhancer is a setting where this can be tested cleanly, because ground truth for the inputs is available.
Core Claim
Let $s \in \mathbb{C}^D$ be the clean STFT frame over $D$ bins. The enhancer outputs a circular complex Gaussian posterior with mean $\mu$ and covariance $\Sigma = \mathrm{diag}(d) + L L^{H}$, where $d$ holds per-bin variances and $L \in \mathbb{C}^{D\times K}$ holds $K \ll D$ shared error directions. For a mel band with weights $w_{bk}$, the band power $m_b = \sum_k w_{bk}|s_k|^2$ has variance
$$ \mathrm{Var}(m_b) = \sum_{k,l} w_{bk} w_{bl}\left(|\Sigma_{kl}|^2 + 2\,\mathrm{Re}(\mu_k^{*}\Sigma_{kl}\mu_l)\right), $$
which follows from the fourth-moment identity for circular complex Gaussians. A diagonal model keeps only the $k=l$ terms. When errors are positively correlated within a band, it therefore underestimates feature variance. The size of the miss grows with the speech mean $\mu$, through the second term, which means voiced frames are hit hardest.
Mel bands span many bins, so the underestimate can be large. Because $\mu$ and $\Sigma$ change frame to frame, no fixed per-band correction can repair it. Our claim is that predicting $L$, and propagating along its columns, gives better-calibrated recognizer inputs and lower WER than diagonal uncertainty at a matched number of forward passes.
Research Program
The enhancer. We train a DPCRN with three output heads: a Wiener-filter mean, per-bin variances $d$, and the factor matrix $L$. Following the harmonic-coupling argument, a few columns of $L$ are parameterised as combs at integer multiples of a pitch estimate predicted by the network itself, with learned amplitude envelopes. A pitch error then moves all harmonics' uncertainty together, roughly a single direction in bin space. A few further columns are banded, covering leakage within roughly the nine-bin span that the meta-adaptive filter work found useful. The remaining columns are left unconstrained.
Training minimises the complex Gaussian negative log-likelihood of the clean frame. Using the Woodbury and matrix-determinant identities, the cost is $O(DK^2)$ per frame instead of $O(D^3)$. A time-domain SI-SDR term guards enhancement quality, as in the approximate-MAP work.
Propagation into the recognizer. We place sigma points at $\mu \pm \sqrt{K}\,L_{:,j}$ (the columns of $L$, each scaled by $\sqrt{K}$) plus the mean, giving $2K+1$ points. The diagonal residual $d$ is handled analytically at the log-mel level with the band-variance formula above. Each sigma point is pushed through the entire acoustic model and scored with the expected-posterior rule $L^{(OU2)}$, the scoring that gave the best CHiME-2 numbers in Uncertainty propagation through deep neural networks. This addresses the dimensionality problem raised for the unscented transform, which normally needs $2D+1$ points: the low-rank factor tells us where the few sigma points should go.
Analysis. We measure calibration directly. For held-out frames we compare the propagated log-mel variance with the empirical squared error against the clean features, split by voiced, unvoiced and silent frames and by SNR. We also report the negative log-likelihood of the clean features and the 95%-interval IOU used by Input uncertainty propagation through trained neural networks.
Baselines
The do-nothing reference is the recognizer decoding the enhancer's mean output as a point estimate. On CHiME-2 the closest reported anchor is the 21.67% WER baseline of Uncertainty propagation through deep neural networks. We re-run it with our own front-end and acoustic model rather than borrowing that number.
The strongest existing method is entire-network Monte Carlo propagation of diagonal input uncertainty, which that paper reports at 21.06% WER on CHiME-2. We implement it on the same DPCRN front-end, but with only the diagonal head trained and the per-bin likelihood of Integrating Uncertainty into Neural Network-based Speech Enhancement. We run it twice: once with the same $2K+1$ forward passes we allow ourselves, and once with a generous budget of several hundred samples. The generous version shows whether diagonal uncertainty is limited by sampling or by its structure. We also include the entire-network unscented transform on diagonal uncertainty, which needs $2D+1$ passes. On AURORA4 we anchor to the relative reductions reported in Propagation of uncertainty through multilayer perceptrons for robust automatic speech recognition. All systems share the training data, backbone size (within 5% of parameters), acoustic model, and an equal hyperparameter-search budget.
Two ablations isolate the key ingredient. The first keeps the low-rank covariance but makes every column unconstrained, removing the harmonic and banded anchoring. This tests whether physical structure matters or rank alone suffices. The second keeps the diagonal model and adds a per-mel-band variance multiplier fitted on development data, possibly separate for voiced and unvoiced frames. This is the cheapest possible repair of the formula's missing off-diagonal terms. We beat the strongest baseline if, at matched passes, structured propagation lowers CHiME-2 WER beyond seed-to-seed noise. We also require it to reduce clean-feature negative log-likelihood in voiced frames, and to do both without lowering enhancement PESQ or SI-SDR on the DNS test set relative to the diagonal model. For enhancement, the reported references are PESQ 2.69 from Uncertainty Estimation in Deep Speech Enhancement Using Complex Gaussian Mixture Models and POLQA 3.06 from the approximate-MAP model.
Why not just…?
Why not just recalibrate diagonal variances with a per-band scale factor? This is the most serious threat. If inter-bin correlation within a mel band were roughly constant, a learned multiplier would capture it, and the covariance head would be unnecessary. The variance formula says the missing term scales with the frame's own speech mean and with where the harmonics fall. That predicts a single multiplier must over-inflate silent frames whenever it corrects voiced ones. The calibration analysis split by frame type tests this prediction directly. If a voiced/unvoiced-conditioned multiplier matches our WER and likelihood, we will report that the cheap fix suffices and that the structured covariance is not worth its cost.
Why not just draw many Monte Carlo samples? Monte Carlo is the most accurate way to propagate a given distribution, and it is our strongest baseline. But it propagates whatever covariance it is handed. Sampling a diagonal posterior a thousand times still produces uncorrelated bin errors. The large-budget diagonal Monte Carlo run separates the two explanations. If it reaches our structured model's WER, the bottleneck was sampling and our claim is false. If it stalls above, the bottleneck was structure. Monte Carlo from our own low-rank posterior is also run, to check that the $2K+1$ sigma points lose little against sampling.
Why not just feed the predicted variances to the recognizer as extra input features and let it learn? A recognizer trained on enhancer means plus variance maps could in principle learn to discount unreliable bands, with no explicit propagation. We train this variant on the same data, giving it both the diagonal head and the factor matrix $L$ as inputs. If it matches structured propagation, then the value lies in the covariance information rather than in propagation, and we will say so. Explicit propagation has one advantage this variant lacks: it keeps working when the enhancer is retrained or replaced without retraining the recognizer. We test that by swapping in a second front-end, a FullSubNet+-style model from FullSubNet+: Channel Attention FullSubNet with Complex Spectrograms for Speech Enhancement, at test time.
Why not just propagate layer by layer, which is cheaper still? The CHiME-2 result in Uncertainty propagation through deep neural networks, where layer-wise propagation raised WER from 21.67% to 23.82%, already rules this out, and we do not revisit it.
Proposed Experiments
The primary benchmark is CHiME-2 Track 2, with WSJ0 speech in real home noise at six SNRs from −6 to 9 dB, scored by WER. It is the setting where entire-network propagation has a reported number to beat. AURORA4 serves as a second recognition benchmark, scored by relative WER reduction. The enhancer is trained on the DNS Challenge corpus described in INTERSPEECH 2021 Deep Noise Suppression Challenge, with recognition-domain fine-tuning. Its stand-alone quality is checked on the DNS test set with PESQ, STOI and SI-SDR, so that gains downstream are not bought with worse enhancement. Because the propagation passes run on the recognizer side, we also report recognizer compute per utterance against the Monte Carlo baselines.
Three checks decide the outcome. First, in voiced frames the diagonal model's propagated log-mel variance should fall clearly below the empirical error, while ours should track it. Second, the per-band multiplier should be unable to fix voiced and silent frames at once. Third, the WER advantage over diagonal propagation should be largest at low SNR and in strongly voiced speech, and close to zero in unvoiced segments. If the first check fails, meaning diagonal variances turn out to be calibrated at the mel level, the premise is wrong and the project stops there.
References
- Propagation of uncertainty through multilayer perceptrons for robust automatic speech recognition
- Uncertainty propagation through deep neural networks
- Integrating Uncertainty Into Neural Network-Based Speech Enhancement
- Uncertainty Estimation in Deep Speech Enhancement Using Complex Gaussian Mixture Models
- Meta-Learning for Adaptive Filters with higher-order Frequency Dependencies
- A Convolutional Neural Network with Non-Local Module for Speech Enhancement
- Variational Speech Separation of More Sources than Mixtures
- DPCRN: Dual-Path Convolution Recurrent Network for Single Channel Speech Enhancement
- Uncertainty-Driven Spectral Compressive Imaging with Spatial-Frequency Transformer
- Input uncertainty propagation through trained neural networks
- Bayesian Analysis of Uncertainty in the GlobCover 2009 Land Cover Product at Climate Model Grid Scale
- Simulated high-latitude soil thermal dynamics during the past 4 decades
- FullSubNet+: Channel Attention Fullsubnet with Complex Spectrograms for Speech Enhancement
- INTERSPEECH 2021 Deep Noise Suppression Challenge