Worst-Case Architecture Search for Streaming Universal Speech Restoration

Real recordings are rarely damaged in only one way. A phone call can be noisy, reverberant, clipped, band-limited and missing packets all at once. Recent "universal" speech enhancement systems, trained to undo all of these together, now handle such mixtures well, but the best of them are large, non-causal server models. Their compute is also spent unevenly: a network can score well on average while quietly failing on the rarer, harder damage types, such as missing bandwidth or lost packets, that need the model to synthesize content rather than suppress it. Meanwhile, hardware-aware neural architecture search, which automatically picks layer types and sizes to fit a measured latency and memory budget on a specific chip, has been developed almost entirely for image classifiers and text Transformers.

We propose to search for causal, streaming speech restorers under hard per-frame latency and peak-memory limits measured on real devices. The search uses an objective that scores each candidate by its worst performance across groups of degradation types, not by its average. The search space mixes three families of operators: time-domain blocks (suited to clipping and packet loss), time-frequency masking blocks (cheap noise suppression) and time-frequency mapping blocks (needed to regenerate destroyed spectrum). The central question is whether a fixed compute budget is best spent differently on different chips and for different damage types. A second question is whether that difference is large enough to beat simply shrinking a strong existing model.

Core Claim

Under a strict streaming budget, the main loss of restoration quality is not a uniform drop. It is concentrated in the degradations that masking cannot fix. A search that averages quality over a mixed training distribution will cut exactly the expensive mapping capacity those cases need. Our claim is that two ingredients together recover much of that loss at equal measured latency, relative to uniformly scaled-down discriminative models and to masking-only searched models:

  • an objective that maximizes the minimum over degradation groups;
  • a search space where masking versus mapping and time versus time-frequency processing are explicit choices.

We further claim that the resulting architectures differ between an ARM CPU and an embedded GPU, enough that swapping them across devices costs real latency.

Formally, let $a$ be a candidate architecture and let $Q_k(a)$ be its normalized restoration score on degradation group $k$ (for example clipping+reverberation, band-limited+codec, packet loss+noise). The score combines downstream word or character accuracy with a non-intrusive quality score. The search solves

$$\max_a \; \min_k Q_k(a) \quad \text{subject to} \quad t_D(a) \le h, \;\; m_D(a) \le M_D,$$

where $t_D(a)$ is the measured per-frame inference time on device $D$, $h$ is the frame hop (so the model never falls behind the audio stream), and $m_D(a)$ is peak working memory against the device budget $M_D$.

Relation to Prior Work

The closest work is UL-UNAS. It uses a reinforcement-learning controller to search convolutional U-Net blocks for real-time denoising, rewarding low multiply-accumulate counts, and reaches PESQ 3.09 on VoiceBank+DEMAND at 35M multiply-accumulates. It differs from our proposal in three ways:

  • It handles only additive noise.
  • Its log-power masking design cannot regenerate zeroed spectrum.
  • Its cost is an arithmetic proxy rather than measured device time.

We reuse its causal time-frequency attention as one candidate operator. Latency-Controlled NAS for Streaming Speech Recognition shows that latency-constrained search works for streaming speech models, but it targets recognition, not signal reconstruction.

Three papers motivate the operator families. EDNet shows, with a hand-designed gate between masking and mapping streams, that multi-distortion enhancement needs both. Its split is fixed by hand and tuned to no device. The multistage URGENT system won URGENT 2024 by processing clipping and packet loss in the time domain and noise and reverberation in the time-frequency domain. It did so by stacking four separate models, which is the redundancy a joint search should remove. TF-GridNet supplies the strongest discriminative design: complex spectral mapping with interleaved full-band and sub-band recurrent layers. We use a causal version of its blocks as candidates.

Four findings constrain the design:

  • Hallucination. The URGENT challenges (URGENT, Interspeech 2025 URGENT) show generative restorers win listener ratings but hallucinate words, and a discriminative sub-band recurrent model won the 2025 limited-data track. We therefore search only discriminative mapping/masking models and make recognition accuracy part of $Q_k$.
  • Data scale. The same 2025 report found no clear gain from 60,000 over 2,500 hours. This justifies searching on the curated 2,500-hour track.
  • Data quality. Lessons Learned from URGENT 2024 documents falsely labeled sampling rates in common "clean" corpora, so we train only on the challenge-curated data.
  • Quality metric. The same paper finds UTMOS and SCOREQ track human ratings while DNSMOS does not, which fixes our choice of quality metric.

The search machinery is borrowed:

  • Latency estimation. HAT contributes weight-sharing supernets, learned latency predictors and evolutionary search, plus the finding that the same layer change can be cheap on a GPU and expensive on an ARM CPU. LitePred and On Latency Predictors for NAS inform the predictor design.
  • Search-space coupling. CompOFA shows that coupling depth and width shrinks the space and reduces interference between subnetworks during supernet training. We couple the width and depth within each operator family the same way.
  • Hard constraints. ConNAS and DUCCIO show that soft latency penalties often produce models that miss the budget. Because we use evolutionary search over measured costs, we can enforce constraints by rejecting infeasible candidates. We do not need to adopt their gradient machinery.

Research Program

Search space. The supernet is a causal streaming restorer with 10–20 ms hops and three stages:

  • Optional time-domain front end. A causal temporal convolution stack whose depth is searched, with zero depth allowed. It targets clipping and packet loss.
  • Time-frequency trunk. Each layer chooses among a depthwise-separable convolution, a causal sub-band recurrent block, a full-band frequency recurrent block, or the folded causal attention from UL-UNAS. Each layer also chooses whether its output head predicts a bounded mask or directly predicts real and imaginary spectral values.
  • Searched output strategy. Either a pure mask, pure mapping, or a learned gate in the spirit of EDNet.

Widths are restricted to multiples friendly to vector units, following The Case for Co-Designing Model Architectures with Hardware.

Cost models. On each target device we measure every operator configuration in streaming mode with its recurrent state carried across frames. We fit a learned predictor, because the full-band and sub-band recurrent layers reshape tensors between time-major and frequency-major layouts, and that likely breaks simple additive lookup tables. We test that assumption directly rather than assume it. Final candidates are always re-measured on the device.

Training and search. The supernet is trained on URGENT 2025 Track 1 data with its seven simulated distortions. Degradation-group labels come for free from the simulation pipeline. Evolutionary search then maximizes the worst-group objective under the hard constraints. The top candidates are retrained from scratch, and we report rank correlation between supernet and stand-alone scores, since weight sharing can scramble rankings.

Targets. A Raspberry Pi-4 ARM CPU, as used by HAT, and an NVIDIA Jetson TX2, as used by ConNAS.

Baselines

Do nothing / naive reference. The unprocessed degraded input, scored on the same per-group metrics. This anchors how much any restorer helps each group, and in particular whether some groups are hurt by enhancement. That matters for recognition accuracy, as Plugin Speech Enhancement shows on noisy ASR.

Strongest existing method, as a quality ceiling. The non-causal TF-GridNet trained on the same data. The URGENT 2024 paper reports PESQ 2.76 and word accuracy 82.87% for this baseline. VoiceFixer is a generative reference that scored NISQA 3.65 but only 82.87%→66.19% word accuracy. Neither is a streaming competitor. They bound what is achievable and show the hallucination failure we avoid. We found no reported result on Lacuna for the on-device streaming latency of TF-GridNet or for any URGENT 2025 system on edge hardware.

Strongest fair competitor: a uniformly scaled causal discriminative model. A causal TF-GridNet-style model, plus a causal sub-band recurrent model resembling the 2025 Track 1 winner. Each is shrunk by a CompOFA-style compound width/depth scaling until it meets the same measured latency and memory budget on each device. It is trained with the same data, epochs and tuning budget as our retrained winners. This is the most dangerous competitor because it keeps a design already proven on this exact benchmark. Beating it means higher worst-group score at equal measured latency on both devices, with mean score no worse by more than seed variance.

Masking-only searched model. UL-UNAS's search space and causal attention, rerun on URGENT data with our measured-latency constraint. This isolates the value of the mapping operators.

Ablations. These isolate the key ingredients:

  • the same search with a mean-over-groups objective instead of the minimum;
  • the same search using multiply-accumulates instead of measured latency;
  • a cascade of separately trained lightweight specialists (time-domain declipper/concealer, denoiser, bandwidth extender) under the same total budget.

All searched variants receive the same number of evaluated candidates and the same retraining budget.

Why not just…?

Why not just shrink TF-GridNet or a sub-band RNN until it fits? This is the real threat. A well-tuned uniform shrink may lose little. HAT's evidence that layer costs differ sharply across devices suggests a uniform shrink wastes budget on at least one platform, but that is evidence from translation models, not audio. The matched-latency comparison settles it per device. If the shrunken model ties our searched model on worst-group score on both devices, we will report that the search adds nothing. A negative result here would itself be a useful finding for edge deployment.

Why not just rerun UL-UNAS's search on composite data? Its masking-only, log-power output cannot restore zeroed bands or dropped frames, since a mask applied to zero stays zero. We therefore expect it to fail specifically on the band-limited and packet-loss groups. The masking-only baseline tests this directly. If mapping heads are added to its space, it becomes our mean-objective ablation, so the remaining difference is exactly the worst-group objective and the measured cost.

Why not just chain tiny specialists? Distortions interact. Reverberation smears the sharp edges that a declipper relies on, and the field's earlier critiques of cascades (SonicMaster, Speech Denoising and Dereverberation Using Probabilistic Models) report error accumulation across stages. The equal-budget cascade ablation measures this, especially on the clipping+reverberation group.

Why not just add a soft latency penalty to differentiable search? ConNAS reports that penalty-based searches such as FBNet and TAS frequently missed hardware budgets where its constraint-guided method met them in 100% of runs. A search that misses a streaming budget produces audio dropouts. Our evolutionary search rejects infeasible candidates outright, so the constraint is exact by construction. The open question is only search efficiency, and we report search cost alongside results.

Proposed Experiments

All models are evaluated on the URGENT 2025 non-blind and blind test sets, with URGENT 2024 as a secondary set. For each degradation group we report character or word accuracy, PESQ, UTMOS and SCOREQ, together with measured per-frame latency, real-time factor and peak memory on both devices.

There are four decisive checks:

  • Worst-group gain. The worst-group search should improve the weakest group over the mean-objective search at equal latency, with mean score within seed noise.
  • Device specificity. Swapping the Pi-found and TX2-found winners should violate or waste budget. This shows device-specific search matters.
  • Beating the shrink. Searched models should beat the uniformly scaled causal discriminative baseline on worst-group score on both devices.
  • Cost-model accuracy. The learned predictor should rank streaming time-frequency candidates better than additive lookup tables, judged by rank correlation against on-device measurement.

We will also inspect where the search spends mapping capacity and the time-domain front end. If those choices track the degradation groups that masking cannot fix, that supports the central claim mechanistically and not just through the scores.