When speech is recorded in a room, the microphone hears the direct voice, then a burst of early echoes from nearby walls within a few tens of milliseconds, then a long, dense tail of later echoes. These stages hurt speech differently. The earliest echoes mostly tint the voice's timbre and can even help listeners. Echoes arriving roughly 70 to 300 ms late smear one sound into the next and do the most damage to speech recognition. Yet the losses used to train dereverberation networks, and the objective scores used to rank them, treat all leftover reverberation as one undifferentiated error. They also cannot distinguish harmless leftover echo from artifacts the network invented. We propose a measurement that splits a system's output error by arrival delay: fit a short linear room filter that maps clean speech onto the output, read off how much residual energy sits at each delay, and treat whatever the filter cannot explain as artifact. We then use a listening test to learn how much humans actually care about each delay band. The same decomposition, weighted by those human-derived weights, becomes a training loss.
The opportunity is concrete. The literature contradicts itself about whether early reflections should be removed. One study found that deleting an early-reflection inverse filter improved a perceptual score and cut compute by 96.7% (On the enhancement of dereverberation algorithms based on a perceptual evaluation criterion). Another reached 7.9% word error rate on REVERB precisely by modeling early reflections explicitly (Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation). Nobody has checked whether this disagreement reflects real perception or the metrics used to judge it. A delay-resolved error measure, validated against listeners, can settle that question, and it gives a training signal that should resist the "fool the quality predictor" failure seen when networks are trained against learned quality models.
Core Claim
Our hypothesis is that standard objective measures misallocate their penalty across reverberation stages. Scale-invariant signal-to-distortion ratio (SI-SDR, a waveform-matching score), PESQ and STOI (standard intrusive quality and intelligibility estimators), and learned non-intrusive quality predictors should over-penalize retained early reflections, which listeners tolerate. They should under-penalize residual energy in the 70–300 ms band, which damages intelligibility, and they should partly confuse generated artifacts with residual reverberation.
We further claim that a loss built from a delay-resolved projection of the output onto clean speech closes much of this gap. Its weights come from human ratings, it treats unexplained components as artifacts, and it does so better than the two cheap fixes a reviewer would suggest. The first fix is moving the training target to "direct plus early reflections" at a fixed cutoff. The second is training against a learned quality predictor.
Research Program
The core tool is a projection computed in the short-time Fourier domain, borrowing the per-frequency linear regression that forward convolutive prediction uses to estimate a room filter (Convolutive Prediction…). We repurpose that regression for evaluation rather than enhancement.
Let $S(t,f)$ be the clean anechoic speech spectrogram and $\hat Y(t,f)$ the system output at frame $t$ and frequency $f$. For each frequency we fit complex filter taps $g_\ell(f)$ over frame lags $\ell = 0,\dots,L$ by ridge-regularized least squares. Whatever the filter cannot explain is the artifact term $E$:
$$\hat Y(t,f) = \sum_{\ell=0}^{L} g_\ell(f)\, S(t-\ell,f) + E(t,f).$$
Each lag corresponds to a physical arrival delay, so the taps can be grouped into bands. Lag 0 is within one analysis window and captures early coloration and direct-path gain. The lags covering roughly 70–300 ms are the smearing band identified by RIR truncation experiments (The harming part of room acoustics in automatic speech recognition). Longer lags are the diffuse tail. The loss is
$$\mathcal{L} = \sum_{k} w_k \sum_{f}\sum_{\ell \in \mathcal{B}_k} \lvert g_\ell(f) - \delta_{\ell 0}\,\bar g(f)\rvert^2 \;+\; \lambda \sum_{t,f} \lvert E(t,f)\rvert^2 .$$
Here $\mathcal{B}_k$ is the set of lags in band $k$ and $w_k$ is that band's weight. The term $\delta_{\ell 0}$ equals 1 only at lag 0, so only the lag-0 tap is compared against $\bar g(f)$, a smoothed target gain. Because only deviations from this smooth gain are penalized, mild spectral tinting at lag 0 costs little. The artifact weight $\lambda$ scales the unexplained energy. The fit has a closed form (it solves the least-squares normal equations), so the loss is differentiable. It needs the clean reference, which simulated training data provides.
This is the audio analogue of what Perceptual Fidelity Aware Mean Squared Error does for images. That work smooths away perceptually irrelevant detail before computing squared error, raising rank correlation with human ratings on TID2008 from 0.7718 to 0.9162. It also mirrors Shift-Tolerant Perceptual Similarity Metric, which builds tolerance for imperceptible shifts into the metric's structure instead of relying on data. Our tolerance is aimed specifically at the early, convolutional part of the error, which has no image counterpart.
The program has three phases.
Phase one: controlled stimuli. Starting from 48 kHz anechoic EARS speech and real room responses in EARS-Reverb (EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation), we truncate and attenuate each room response band by band to create outputs with known residual energy per band. We then add controlled artifacts drawn from real system failures: over-suppression, spectral-subtraction musical noise, and outputs of a network trained against a learned quality predictor.
Phase two: listening test. We run a MUSHRA-style test, in which listeners score several versions of a clip against a hidden reference. We collect quality ratings and a separate intelligibility measure through transcription and ASR word error rate. Collecting both matters because quality and intelligibility respond differently to processing (Study on the Correlation between Objective Evaluations and Subjective Speech Quality and Intelligibility). Regressing human scores on the band energies and artifact energy yields the weights $w_k$ and $\lambda$. The same regression applied to each objective metric reveals that metric's implied weighting, which puts the claimed misallocation on a common scale. P-Reverb: Perceptual Characterization of Early and Late Reflections for Auditory Displays motivates this design: it found that listeners are much more sensitive to room changes when the late tail is present. However, it studied rendering rather than restoration, so we cannot simply import its thresholds.
Phase three: training. We train the same enhancement networks with the resulting loss and compare them against the alternatives below.
Baselines
The naive reference points are two. The first is the unprocessed reverberant input. The second is a network trained with plain SI-SDR against the anechoic target, which is how WHAMR!-style systems are usually trained (WHAMR!: Noisy and Reverberant Single-Channel Speech Separation). Any stage-aware method must beat the second on human quality without losing word error rate.
The strongest predictive competitor is forward convolutive prediction. It estimates direct speech with a first network, fits a linear room filter, and refines the result with a second network (Convolutive Prediction…). That paper reports 7.9% word error rate on REVERB and 7.5 dB SI-SDR on WHAMR!. It is the strongest competitor because it already exploits the same linear-convolution structure we use, but inside the enhancer rather than the loss. We will retrain its architecture unchanged under three losses: its original loss, our loss, and the fixed-cutoff alternative. The network, data, epochs, and hyperparameter search budget will be identical across the three. Reproducing roughly the reported REVERB number with the original loss is a precondition for the comparison. Beating it means a significant gain in listener MUSHRA score on EARS-Reverb and on REVERB real recordings, at equal or lower word error rate.
The strongest perceptual competitor is SGMSE+, a diffusion model, meaning a generator that learns to turn noise into clean speech through repeated denoising steps. On EARS, as reported in the EARS benchmark paper, it reached a POLQA score of 3.40 against a 1.71 noisy baseline and cut word error rate from 33% to 16%. It also ran about 170 times slower than Conv-TasNet (EARS…). The file does not state whether these figures are for the reverberant subset. We will run the released model ourselves on the identical EARS-Reverb split. A predictive network with our loss "beats" it if it matches its listener score within the confidence interval at a small fraction of its runtime.
We also include a learned-quality-supervision baseline following Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment. Its predictor, Uni-VERSA-Ext, reaches a linear correlation coefficient of 0.97 with human ratings on URGENT24, and its training includes that paper's feature-matching and self-supervised regularizers. No reported result found on Lacuna for this method on a dereverberation benchmark.
Two ablations isolate the key ingredients. The first sets all $w_k$ equal, testing whether human-derived weighting matters beyond the projection. The second sets $\lambda$ equal to the reverberation weights, testing whether separating out artifacts matters.
Why not just…?
Why not just train on the direct-plus-early target at a fixed cutoff, as the extended ratio mask does? Ideal Ratio Mask Estimation Using Deep Neural Networks for Monaural Speech Segregation in Noisy Reverberant Conditions already showed that keeping reflections under 50 ms in the target raises PESQ and STOI. This is the most serious threat. A cutoff sweep at 12.5, 25, and 50 ms might capture most of the gain. The difference is structural. A fixed target still demands exact waveform reproduction of the early echoes. It weights the harmful 70–300 ms band the same as the diffuse tail. It gives no separate penalty for invented artifacts. We will run the cutoff sweep with the best cutoff chosen on validation data. If it matches our loss on listener scores and word error rate, we will report that the benefit comes from target choice alone and that the loss adds nothing.
Why not just train against a learned quality predictor? Doing so led networks to produce non-speech artifacts that scored well. Regularizers were needed to stop this, and even then character error rate rose (Improving Speech Enhancement with Multi-Metric Supervision…). Our artifact term directly penalizes energy that cannot be written as a filtered copy of clean speech, which is exactly the shortcut that paper describes. The phase-two stimuli include this baseline's outputs, so we can check whether its predictor rates them above what listeners give them.
Why not just use late-only suppression such as spectral subtraction, since removing the early stage already helped? The 6–9% QAreverb gain reported by Prego et al. was measured with a single objective metric on 204 signals (On the enhancement…). Later refinements of the same pipeline were scored only with Allen's perceptual score (Perceptual improvement of a two-stage algorithm for speech dereverberation). Meanwhile, echoes in the 70–300 ms band hurt recognition. Our decomposition tells us, per band, whether a late-only system leaves harmful mid-delay energy behind. The listening test shows whether listeners agree with the metric that favored it.
Proposed Experiments
Check one: the premise. On the controlled stimuli, we compare the band weighting implied by each objective metric with the human weighting. If PESQ, STOI, SI-SDR, and the learned predictor already track human ratings across bands, with no significant difference in fitted weights, the misallocation claim is false and the loss is unnecessary. We also test whether the delay-resolved score correlates better with listeners than each existing metric on held-out rooms. For this we use Spearman correlation, the measure PAMSE used, so the comparison is on the same footing.
Check two: the paradox. We reimplement the two-stage pipeline (inverse filtering followed by spectral subtraction) and the subtraction-only variant on EARS-Reverb and have listeners rate both. If listeners prefer the two-stage version while objective metrics prefer subtraction-only, the earlier finding is at least partly a metric artifact. If both prefer subtraction-only, it reflects real perception.
Check three: training. We train all systems described in Baselines on EARS-Reverb and on WHAMR!'s reverberant condition and evaluate on REVERB real recordings, reporting listener MUSHRA, ASR word error rate, and the standard metrics. The proposal succeeds only if our loss beats the best fixed cutoff on listener quality at equal word error rate, and shows less artifact energy than learned-predictor training. The loss must also transfer to real recorded rooms, not only simulated ones.
References
- On the enhancement of dereverberation algorithms based on a perceptual evaluation criterion
- Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation
- The harming part of room acoustics in automatic speech recognition
- Perceptual Fidelity Aware Mean Squared Error
- Shift-Tolerant Perceptual Similarity Metric
- EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation
- Study on the Correlation Between Objective Evaluations and Subjective Speech Quality and Intelligibility
- P-Reverb: Perceptual Characterization of Early and Late Reflections for Auditory Displays
- WHAMR!: Noisy and Reverberant Single-Channel Speech Separation
- Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment
- Ideal Ratio Mask Estimation Using Deep Neural Networks for Monaural Speech Segregation in Noisy Reverberant Conditions
- Perceptual improvement of a two-stage algorithm for speech dereverberation