Small speech-enhancement networks for hearing aids and earbuds spend the same compute on every part of the spectrogram, even though in any given 10 ms frame most time-frequency regions contain only noise. Classical signal processing has long had a cheap, well-understood estimate of where speech is: the speech presence probability (SPP), a per-frequency number between 0 and 1 computed from running noise estimates. We propose to use that estimate as a router. In each frame, a statistical filter costing almost nothing processes all frequency bands, and a searched neural block runs only on the K bands most likely to contain speech. Because K is fixed, every frame costs the same. The method can therefore be scheduled on a microcontroller like any static network, and the network's capacity goes where the speech is.
The second ingredient is hardware-aware neural architecture search, meaning automatic design of a network's layers using measured on-device latency and memory rather than operation counts. The search chooses the neural block's topology, the number of routed bands K, and the cheap path's form, all at once, under a hard worst-case latency and memory limit measured on the target chip. The question we want to answer is narrow and falsifiable. At identical measured worst-case latency and peak memory, does a network that reallocates compute by SPP beat the best static network the same search can find?
Core Claim
Two existing threads do not yet meet. The first is dynamic computation in speech enhancement. Dynamic Gated Recurrent Neural Network for Compute-efficient Speech Enhancement skips updates for low-activity recurrent neurons and keeps about baseline quality down to 50% updates. It gates on internal hidden-state magnitudes, not on acoustic evidence. Its authors also note that threshold-based skipping produces variable load that is hostile to real-time scheduling. The second thread is neural architecture search for enhancement. UL-UNAS searches a lightweight U-Net with frequency-dependent activations and reaches PESQ 3.09 on VoiceBank-DEMAND at 35M multiply-accumulates. Its search is guided by multiply-accumulate counts rather than device measurements, and its graph treats all bands uniformly.
Our claim is that a fixed-budget, SPP-ordered allocation of neural capacity turns the classical speech/noise prior into something a static scheduler can execute. The per-frame cost is
$$C = C_{\text{spp}} + C_{\text{cheap}}(G) + K\,c_{\text{heavy}}(a) + C_{\text{gather}}(K) \le T_{\text{hop}},$$
where $G$ is the number of frequency bands, $K \le G$ is the number routed to the neural block, $a$ is the searched block architecture, $c_{\text{heavy}}(a)$ is its measured per-band cost, $C_{\text{gather}}$ is the measured cost of collecting and scattering the selected bands, and $T_{\text{hop}}$ is the frame hop.
Because every term is input-independent, the worst case equals the typical case. That property separates this design from data-dependent gating. It also differs from the closest searched model, UL-UNAS, in three ways: it routes on an acoustic signal, it searches the split of work between statistical and neural processing, and it constrains measured latency rather than operation counts.
Research Program
Cheap path. This path is a single-channel SPP-weighted Wiener gain built from two classical estimators. The noise tracker of A noise estimation method based on speech presence probability and spectral sparseness uses fixed priors and a sparseness measure so that narrowband noise bursts do not inflate SPP. The smoothing of Improved a posteriori speech presence probability estimation based on cepstro-temporal smoothing and time-frequency correlation suppresses isolated SPP outliers. Both supply the method. We reuse them rather than invent an estimator. The gain rule follows the SPP-scaled attenuation of Codebook-based speech enhancement using Markov process and speech-presence probability, where low SPP gives strong attenuation and high SPP relaxes it.
Router. Frequency bins are grouped into roughly 32 perceptually spaced bands. Each frame, bands are ranked by smoothed mean SPP and the top K go to the neural path. Hysteresis keeps band membership from flickering between frames. This guards against the musical-noise artifacts that hard spectral switching is known to cause.
Neural path. Searched blocks operate on the selected bands. Each block has a recurrent state per band, and when a band is not selected its state is carried forward, as in the neuron-skipping of DG-RNN but at band granularity. The block emits a residual correction to the cheap gain, so a routing miss falls back to classical filtering rather than to no filtering. The neural path is optionally trained with SPP as an auxiliary target, following Multi-task Single Channel Speech Enhancement Using Speech Presence Probability As A Secondary Task Training Target. That paper reports a reverberant-set ΔPESQ gain from 0.14 to 0.24 from adding the SPP task.
Search space. The block choices come from UL-UNAS's convolution and inverted-residual families and its causal time-frequency attention. To these we add K, band-group widths, and cheap-path variants: a pure Wiener gain, or a Wiener gain plus a tiny per-band correction.
Search procedure. We train a weight-sharing supernet in the manner of HAT and run multi-objective evolutionary search as in MoGA. Candidates are scored against a latency and memory look-up table measured on the device. The table includes the gather/scatter operations and the STFT overlap-add buffers, which vision-oriented tables omit. Hard limits are enforced by rejecting infeasible candidates. We also compare against the constraint-steering of ConNAS, which reports satisfying constraints in 100% of runs. The motivation for measuring rather than counting comes from HAT and MoGA, which show that equal operation counts can produce very different device latencies.
Targets. We target two devices: a Cortex-M-class microcontroller and the GAP8 RISC-V chip used by DUCCIO. DUCCIO's memory-hierarchy-aware search is the precedent for treating SRAM fit as a first-class constraint. The streaming setting uses a 20 ms window and 10 ms hop. Algorithmic latency is held at or below 25 ms, the figure RT-SEMamba targets.
Known risk. A Probability Weighted Beamformer for Noise Robust ASR found that probability-weighted filtering helped on real data but slightly hurt simulated data, because SPP errors distorted speech when the noise was benign. This result constrains the design. The residual-correction formulation and hysteresis exist to contain it, and the per-SNR breakdown below is meant to expose it if they fail.
Baselines
Naive reference. The SPP-weighted Wiener filter alone, with K = 0 and no neural path. It sets the floor and costs almost nothing on device. Any routed model that barely beats it has not justified its neural block.
Strongest published lightweight model. UL-UNAS reports PESQ 3.09 on VCTK-DEMAND at 35M multiply-accumulates and 171k parameters. We will retrain it with our data pipeline and STFT settings and deploy it on both target devices, which UL-UNAS itself did not do. No reported on-device latency was found on Lacuna for UL-UNAS or for GTCRN, which UL-UNAS lists at 34M multiply-accumulates. For context at higher cost, RT-SEMamba reports PESQ 3.18 at 1.05M parameters, measured only on a desktop GPU, and MP-SENet reports 3.50 at 2.05M parameters. These sit outside a microcontroller budget and will be reported only as quality ceilings.
Most important comparison: static search under the same budget. We run our own search with routing disabled, so every band passes through the neural path. It uses the same supernet training, the same look-up table, the same number of evaluated candidates, the same training epochs, and the same hard limits. The two arms differ only in whether SPP reallocates capacity.
Ablation 1: random routing. The same K, with bands chosen at random each frame. This tests whether ordering by speech presence matters at all.
Ablation 2: learned router. The same K, with bands ranked by a tiny learned scorer of matched cost. This tests whether the classical SPP prior adds anything beyond a cheap trainable selector.
Ablation 3: oracle routing. Bands ranked by true SPP computed from clean speech. This gives the upper bound on what routing can deliver.
Ablation 4: band-level DG-RNN. A fixed-fraction version of DG-RNN's Top-A gating applied to the static searched network at matched compute. This isolates acoustic routing from hidden-state routing.
What counts as winning. Beating the static-search baseline means a PESQ gain of at least 0.1 and no loss in ESTOI. The gain must hold at equal or lower measured worst-case frame latency and equal peak SRAM, on both devices, and the latency jitter must be indistinguishable from the static model's.
Why not just…?
Why not just run hardware-aware search on a static network? This is the real threat. If speech energy is spread widely enough, or if static compute is already cheap per band, a static network of equal worst-case cost may match the routed one. The static-search arm decides this directly under an identical search budget. If it ties, we will report that SPP routing is unnecessary at this scale, which is itself worth knowing for hearing-aid designers. We expect routing to help most at low K budgets and at low SNR with sparse speech, so we report the full Pareto curve rather than a single point.
Why not just use dynamic neuron skipping like DG-RNN? DG-RNN reduces arithmetic but does not reduce peak memory, which is set by the full layer widths. Its own authors flag scheduling difficulty for threshold variants. Even its fixed-fraction Top-A variant ignores acoustic evidence and was never profiled on edge hardware. Ablation 4 compares it with SPP routing at matched compute and measured latency.
Why not just learn the router end to end? A learned scorer may fit training noise better than classical SPP and generalize worse. Alternatively, it may simply win. Ablation 2 settles this. We evaluate it on the DNS3 test set and on noise types held out from training, where a classical noise tracker has no training distribution to overfit.
Why not just distill a large model into a small static one? RT-SEMamba shows distillation lifts a one-layer student from 3.06 to 3.18 PESQ. Distillation is orthogonal to routing, so we apply the same distillation recipe to both the static and routed arms. Applying it to only one arm would confound the comparison.
Proposed Experiments
Data. We train on VoiceBank-DEMAND and on the DNS3 training data. Evaluation covers four test conditions. The first is the VoiceBank-DEMAND test set, scored with PESQ, CSIG, CBAK, and COVL. The second is the DNS3 test set; no reported DNS3 number was found on Lacuna for UL-UNAS. The third is stress sets at −5, 0, and 5 dB SNR built as in DG-RNN. The fourth is unseen-noise mixtures.
Hardware measurements. On both devices we measure worst-case and 99th-percentile frame latency, peak SRAM, flash size, and energy per frame.
First decisive check. Plot the routed model's Pareto front against the static-search front on measured latency, not operation counts. Routing is worthwhile only if the fronts separate by at least the 0.1 PESQ margin defined above.
Second decisive check. Order the four routing variants. If the method works as claimed, the gap between oracle and SPP routing should be small and the gap between SPP and random routing should be large. If SPP falls between random and learned routing, the classical prior's value is limited to robustness, and the unseen-noise results will show whether even that holds.
Third decisive check. Break results down by input SNR to test for the benign-noise distortion reported by Bu et al.. A regression at high SNR relative to the static model would show that the residual fallback did not contain routing errors.
Fourth decisive check. Compare listening-test and CBAK scores near band-routing boundaries. A drop there would show that hysteresis did not prevent musical-noise artifacts.
References
- Dynamic Gated Recurrent Neural Network for Compute-efficient Speech Enhancement
- UL-UNAS: Ultra-Lightweight U-Nets for Real-Time Speech Enhancement via Network Architecture Search
- A noise estimation method based on speech presence probability and spectral sparseness
- Improved a posteriori speech presence probability estimation based on cepstro-temporal smoothing and time-frequency correlation
- Codebook-based speech enhancement using Markov process and speech-presence probability
- Multi-task Single Channel Speech Enhancement Using Speech Presence Probability As A Secondary Task Training Target
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
- MoGA: Searching Beyond Mobilenetv3
- Constraint-guided Hardware-aware NAS through Gradient Modification
- Enhancing Neural Architecture Search With Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
- RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation
- A Probability Weighted Beamformer for Noise Robust ASR
- MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra