Speech enhancement networks do better when they let each frequency bin "see" other bins. The reason is physical. A voiced sound puts energy at its pitch and at every integer multiple of it, and a room smears energy across neighbouring bands as it decays. These models are trained almost entirely on simulated reverberation, though, and simulators get some of those couplings right and others wrong. When dry speech recorded in a studio is convolved with a simulated room response (the impulse response, or RIR, which describes how a room echoes a click), the harmonic structure is real, because it comes from the human voice. The room's behaviour is synthetic. Geometric simulators also get it most wrong below roughly 1,400 Hz, where diffraction and standing room modes dominate (Synthetic Wave-Geometric Impulse Responses for Improved Speech Dereverberation).
Our claim is that cross-frequency modelling should be split along this line. One path couples bins at harmonic multiples of the current pitch. Its statistics come from real speech, so it should transfer from simulation to reality almost unchanged. A second path couples adjacent and low-frequency bands to model room leakage and modes. That path inherits the simulator's errors, so it is the only one we should spend scarce real data on. The model is pretrained on simulation and then fine-tuned only on its room path, using a handful of measured real rooms. We predict this closes more of the real-room gap than full fine-tuning, or than fine-tuning a size-matched generic subset. We also predict that undifferentiated global frequency mixing carries low-band simulator errors into the well-simulated high bands.
What exists and what is missing
Several architectures already model cross-frequency structure explicitly, but none asks which of the learned couplings are trustworthy after transfer.
- DPCRN runs a bidirectional recurrent network along the frequency axis within each frame.
- CFTNet learns a global frequency transformation matrix over complex spectra.
- FullSubNet+ fuses a full-band context model with sub-band estimators and attention over bands.
- A Convolutional Neural Network with Non-Local Module for Speech Enhancement uses attention over all time-frequency positions.
The non-local paper also constrains our design. Stacking more than two non-local blocks hurt quality, which is evidence that unrestricted long-range frequency mixing can damage local spectral continuity. Our split replaces "mix everything" with two physically targeted paths.
Two papers come closest to this intersection and each leaves half of it open.
- Speech dereverberation constrained on room impulse response characteristics adds a 3-band crossband model and an energy-decay physics loss to FullSubNet. It trains and tests only on pyroomacoustics shoebox rooms, never on real recordings. Its mismatched-RIR gain is 1.60 to 1.61 WB-PESQ. - Synthetic Wave-Geometric Impulse Responses supplies the motivating evidence. It shows the sim-to-real gap is frequency-localised: the Pearson correlation between low-frequency energy and the gains from wave-simulated RIRs is 0.91. It then uses a plain 2D CNN with no explicit cross-frequency mechanism.
Meta-Learning for Adaptive Filters with Higher-Order Frequency Dependencies supplies our room-path primitive. It uses overlapping "banded" groupings of neighbouring bins and reports +3.21 dB SI-SDR over a Kalman filter on synthetic echo data, but it too never leaves simulation.
On the transfer side, SynthCLIP and Contrastive Learning from Synthetic Audio Doppelgängers motivate the two-stage schedule. SynthCLIP found that synthetic pretraining followed by real fine-tuning beat training on the full real set by 4.4%. The doppelgänger work found that synthetic data generated by varying causal physical parameters transfers well to real audio. Both work on classification and adapt the whole network. We study dense spectral reconstruction and adapt a physically chosen part of it.
Core Claim
In a dereverberation network pretrained on simulated rooms, the cross-frequency couplings that encode source harmonics transfer to real rooms, while the couplings that encode room leakage and low-frequency modes do not. Making these two families separate, explicit modules has two effects. It stops low-band simulator error from contaminating well-simulated bands. It also lets a few real rooms correct the model by updating only the room path.
Research Program
The work starts with a diagnostic that tests whether the premise holds before anything new is built. We train DPCRN, FullSubNet, and a local CNN on geometric and on wave-geometric RIRs, then evaluate on real measured RIRs from MIT IR, BUTReverb and Aachen convolved with held-out dry speech. We break the error down by frequency band. We also compute a cross-band influence map: the gradient of each output band with respect to each input band, averaged over utterances. Contamination is the hypothesis that real-room error in the high band (above 1,400 Hz, which geometric simulation models reasonably well) is larger for global-mixing models than for the local CNN. If so, the excess should track how strongly each high-band output depends on low-band inputs. If contamination is absent, the "protect the high bands" half of the claim fails. Only the selective-adaptation half would then remain to be tested.
The model has two coupling paths on top of a shared encoder and a causal temporal recurrence.
The harmonic path takes a frame-level pitch estimate $f_0(t)$ for frame $t$, from a lightweight pitch tracker run on the input. It gathers the feature vectors at bins nearest $k f_0(t)$ for harmonics $k = 1, \dots, K$, each with a ±1-bin window. A small recurrent model runs along the harmonic index $k$, and the outputs are scattered back to the same bins. Unvoiced frames bypass this path.
The room path is a banded complex-valued recurrence over overlapping groups of neighbouring bins, in the style of Wu et al. It is given wider bands and more capacity below 1,400 Hz, where room modes live.
The two paths are summed through a learned gate per bin. All variants are matched in total parameter count.
Training is two-stage. Stage one pretrains on wave-geometric RIRs from the Guided Wave-Acoustics (GWA) dataset convolved with LibriSpeech and WSJ0. Stage two fine-tunes on a small set of real rooms, with budgets of 5, 20 and 50 measured RIRs. The default is to update only the room path. Test rooms are always disjoint from fine-tuning rooms.
Baselines
The naive reference is the unprocessed reverberant input, alongside each architecture trained on simulation only with no real fine-tuning. These anchor how much of the gap any adaptation recovers.
The strongest prior system on this exact question is the hybrid-simulation pipeline of Synthetic Wave-Geometric Impulse Responses. Its reported numbers are a 28.2% SRMR improvement on real RIRs for hybrid over geometric synthetic-only training, and an 8.3% SRMR improvement when real data is added to training. We retrain SkipConvNet on our exact data splits rather than comparing to the published figures, because our test rooms differ.
We also run DPCRN and FullSubNet with the 3-band crossband model of Bahrman et al., since these are the strongest explicit cross-frequency competitors. We found no reported result on Lacuna for DPCRN, FullSubNet, or CFTNet dereverberation on real RIR benchmarks. Bahrman et al.'s 1.61 WB-PESQ is on synthetic mismatched rooms only and is not comparable.
Every baseline gets the same pretraining data and step count, the same real fine-tuning rooms, and the same hyperparameter search budget (an equal number of trials over learning rate and fine-tuning steps). Each baseline also gets both full fine-tuning and its best parameter-efficient variant.
Three ablations isolate the key ingredient:
- Unconstrained attention: replace the harmonic path with frequency attention of equal size.
- Fixed random subset: fine-tune a size-matched random subset of parameters instead of the room path.
- Inverted split: fine-tune only the harmonic path.
Beating the baselines means two things at the 20-room budget. First, higher SI-SDR and WB-PESQ on held-out real rooms than the best fine-tuned baseline, with a margin exceeding the across-seed standard deviation over three seeds. Second, lower high-band error than global-mixing baselines in the simulation-only condition.
Why not just…?
Why not just train an existing cross-frequency model on wave-geometric RIRs? This is the real threat, since it fixes the gap at its source in the data. We do not compete with it. GWA is the pretraining data in every arm. What remains contested is whether wave simulation leaves a residual gap; the file reports one in The Acoustic Simulation to Reality Gap, from transducer directivity, furniture scattering and irregular boundaries. The second question is whether targeted adaptation corrects that residual more efficiently. If DPCRN on GWA with no fine-tuning matches our fine-tuned model on held-out real rooms, the proposal's contribution reduces to the diagnostic, and we will report it that way.
Why not just fully fine-tune on the real rooms? With 5 to 20 rooms, full fine-tuning risks memorising those rooms and disturbing harmonic couplings that were already correct. The evidence that full adaptation helps (SynthCLIP, and the 28% production WER gain in Bo Li et al.) comes from regimes with far more real data. The room-budget sweep settles this directly. We expect full fine-tuning to catch up by 50 rooms, and the interesting result is the low-budget regime.
Why not let global frequency attention learn harmonics implicitly? The non-local results show that unrestricted mixing over-smooths when stacked. Implicit couplings also cannot be selectively frozen, because they are not separable by physical origin. The unconstrained-attention ablation, together with the cross-band influence maps, shows whether attention learns harmonic structure that transfers. If it matches our model under room-path-equivalent fine-tuning, the explicit pitch indexing is unnecessary.
Proposed Experiments
The decisive checks run in order. First, the contamination diagnostic on real RIRs across three architectures and two simulators. Second, the main comparison at 5, 20 and 50 real fine-tuning rooms. Clean references exist for this setting, so we report SI-SDR, WB-PESQ and per-band log-spectral distance, plus SRMR, which needs no reference. Third, a fully real test on VOiCES distant-microphone recordings, which have no clean target. There we report SRMR and DNSMOS, plus word error rate from a fixed off-the-shelf recogniser on the dereverberated audio. The anchor is the corpus's reported degradation from 9.3% WER on clean speech to 19.0% on distant microphones (VOiCES).
The proposal is supported if three things hold. The harmonic-path ablations show that fine-tuning harmonics alone recovers little. Room-path-only fine-tuning beats the random-subset control. The high-band error gap between global and split models shrinks after room-path adaptation. If the inverted split works as well as the intended one, the physical-origin story is wrong.
References
- Synthetic Wave-Geometric Impulse Responses for Improved Speech Dereverberation
- DPCRN: Dual-Path Convolution Recurrent Network for Single Channel Speech Enhancement
- CFTNet: Complex-valued Frequency Transformation Network for Speech Enhancement
- FullSubNet+: Channel Attention Fullsubnet with Complex Spectrograms for Speech Enhancement
- A Convolutional Neural Network with Non-Local Module for Speech Enhancement
- Speech dereverberation constrained on room impulse response characteristics
- Meta-Learning for Adaptive Filters with higher-order Frequency Dependencies
- SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?
- Contrastive Learning from Synthetic Audio Doppelgängers
- VAST: The Virtual Acoustic Space Traveler Dataset
- Acoustic Modeling for Google Home.
- Voices Obscured in Complex Environmental Settings (VOiCES) Corpus