Scoring Speech Architectures by How Far Back They Actually Listen

Architecture search tries many candidate networks and keeps the best, and most of its cost goes into training each candidate long enough to tell good from bad. "Zero-cost proxies" try to skip that training. They score an untrained network from one forward and backward pass on a single batch, and on image benchmarks they rank networks well enough to cut search cost by about three orders of magnitude. On audio and other time series they mostly stop working: NAS-Bench-360 reports near-zero and highly volatile rank correlations once these proxies leave vision. A plausible reason is that every existing proxy looks at the network as a static function of one input. None of them asks the question that decides most speech architectures: how much does the output at frame $t$ depend on frames far in the past (and, in streaming models, in the future)?

We propose a proxy that measures exactly that, at initialization, on real speech. For each candidate we compute the effective temporal influence profile: how strongly each output frame responds to perturbations of input frames at each time lag, averaged over a batch of utterances. This profile is a single backward pass per probed lag band, so it is still cheap. It captures what a network's nominal receptive field cannot: recurrent state decay, dilation gaps, attention spread at initialization, and normalization effects. The same profile also gives the model's algorithmic lookahead for free. Our claim is that this profile, combined with a standard trainability signal, ranks speech and sequence architectures much better than existing proxies, and that the gain comes from the measured profile, not from simply counting receptive field or FLOPs.

Core Claim

Let $x_{1:T}$ be an input feature sequence and $y_{1:T}$ the network's output at initialization. We define the influence at lag $k$ as

$$ I(k) = \mathbb{E}_{x,\,t}\left[\left\lVert \frac{\partial y_t}{\partial x_{t-k}} \right\rVert_F\right], \qquad k \in \{-K_f,\dots,0,\dots,K_p\}, $$

where the expectation is over utterances from the target corpus and over output positions $t$. $K_p$ and $K_f$ are the past and future horizons probed, and $\lVert\cdot\rVert_F$ is the Frobenius norm. In practice we estimate $I(k)$ with random output projections (vector–Jacobian products), so it costs a handful of backward passes.

From the normalized profile $\hat I(k) = I(k)/\sum_j I(j)$ we derive three quantities:

  • Effective past reach: the lag that contains 90% of past influence mass.
  • Effective lookahead: the same quantity for negative lags.
  • Profile flatness: the entropy of $\hat I$ over past lags.

The proxy concatenates these with a trainability term; we use synflow, which Zero-Cost Proxies for Lightweight NAS found to be the most robust single proxy across vision, NLP and ASR. These features feed a small ranking model.

The specific new claims are threefold. First, effective reach measured on data diverges substantially from analytic receptive field across real speech search spaces. Second, that divergence predicts final error. Third, the profile improves ranking specifically among the top-performing decile, where EZNAS reports that current proxies lose discrimination.

Relation to Prior Work

Three groups of papers play different roles here.

Papers that motivate the gap. NAS-Bench-360 documents the collapse of zero-cost proxies on temporal and audio tasks, but builds no temporal proxy. NAS-Bench-Suite-Zero shows that feeding 13 existing proxies into a surrogate improves rank correlation by up to 42%, but none of those 13 measures temporal dependence. LPZero and EZNAS evolve proxy formulas automatically. Their primitives are layer tensors from a fixed block template, so they cannot express a lag-resolved quantity. Our proxy is hand-designed from one physical question rather than searched.

Papers whose architectures show the target signal. Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search found that the best layer-heterogeneous Conformers widen their kernels in deeper layers. Lightweight NAS for TCNs at the Edge gains large compression by pruning the oldest kernel taps and dilations layer by layer. Both suggest that temporal reach, allocated across depth, is what search is really choosing. Both pay with super-network or seed-network training, which our proxy avoids. Those costs are 23.2 GPU hours on AISHELL-1 for the Conformer search, and hundreds of seed epochs for the TCN pruning.

A paper that constrains the design. Latency-Controlled NAS for Streaming Speech Recognition shows that unconstrained search exploits future frames. It enforces streaming limits through hard causal masks. Our lookahead estimate turns that into a measured quantity that can filter candidates before any training. Because it is measured rather than imposed, it can detect when nominal lookahead is not actually used.

Research Program

We first build the profile estimator for three architecture families: cell-plus-LSTM networks, dilated TCNs, and Conformer blocks. For each family we validate it on a few trained models by checking that the profile at initialization correlates with the profile after training. If it does not, the proxy rests on nothing.

We then assemble ground truth across three architecture families:

  • NAS-Bench-ASR. NAS-Bench-ASR already provides 8,242 TIMIT architectures with phoneme error rate (PER) at 5, 10 and 40 epochs across three seeds, plus measured latency on a 1080Ti and a Jetson Nano.
  • Layer-heterogeneous Conformers. We add a new set of about 200 Conformers drawn from the search space of the Conformer NAS paper: heads in {4, 8, 16}, kernels in {7, 11, 15}, dilation in {1, 2}, FFN width in {256, 512, 1024}. Each is trained to a fixed schedule on AISHELL-1 with two seeds. This is the heterogeneous, attention-bearing space that NAS-Bench-ASR lacks.
  • TCN seeds. A third set of about 300 TCN variants comes from PIT-style seeds on Speech Commands v2 and the 1D NAS-Bench-360 tasks ECG and Satellite.

All proxies, including ours, are computed on every architecture with an identical batch and hardware. Finally, we run proxy-guided regularized evolution on the Conformer space and compare the discovered models to differentiable search.

Baselines

Do-nothing reference. This is the uniformly stacked Conformer, whose reported AISHELL-1 character error rate (CER) is 8.3% in the Conformer NAS paper. For ranking experiments the equivalent naive point is random ordering, together with ranking by parameter count alone. NAS-Bench-Suite-Zero found FLOPs to be among the most robust "proxies", so any new proxy must clear it.

Analytic receptive field plus FLOPs and parameters. This is the ablation that isolates our key ingredient. It uses the same surrogate model and the same features, except that the measured profile is replaced by the receptive field and lookahead computed from the architecture definition. If it matches our proxy, the measurement adds nothing.

Strongest existing training-free method. This is the NAS-Bench-Suite-Zero recipe: all 13 standard proxies plus FLOPs as features to an XGBoost surrogate, trained on the same number of labeled architectures as ours. That paper reports up to a 42% Spearman improvement from this ensemble on its own spaces. We found no reported result on Lacuna for any zero-cost proxy's rank correlation on NAS-Bench-ASR specifically, so we compute it ourselves. Beating this baseline means a statistically reliable gain in Kendall tau, overall and within the top decile, at equal labeled-set size.

Low-fidelity training. NAS-Bench-ASR's 5-epoch and 10-epoch PER columns rank the same networks. They cost real training, which we charge at measured wall-clock time.

Strongest end-to-end search. This is DARTS with the Dynamic Search Schedule from the Conformer NAS paper. It reports 7.5% CER on AISHELL-1 with about 28.9M parameters at 23.2 GPU hours, versus 322 GPU hours for random search. We rerun it on our hardware with the same data and features. Our proxy-guided evolution counts as beating it if it matches 7.5% CER at a parameter count no larger, using less total wall-clock time including final training. Matching within 0.2 CER at a fraction of the cost also counts. Uniform Path Dropout from NAS for Speech Emotion Recognition is included as a second stabilized differentiable baseline.

Why not just…?

Why not just use synflow, which already transfers to speech? Synflow is our trainability component. The open question is whether the profile adds to it. This is a real threat on NAS-Bench-ASR, whose cells are wrapped in a fixed LSTM and therefore vary less in temporal reach than heterogeneous Conformers. We therefore expect our largest gains on the Conformer and TCN sets. If the profile adds nothing beyond synflow on those sets, the idea is wrong.

Why not just compute receptive field and lookahead from the architecture? This is the most dangerous alternative, because it is free and exact in nominal terms. Our bet is that nominal reach overstates real reach, through LSTM forgetting, gaps in dilated stacks, and near-uniform attention at initialization. We also bet that the overstatement varies across candidates in ways that track final error. The analytic-feature ablation in Baselines settles this directly. We also report the per-architecture gap between nominal and effective reach, so readers can see whether the gap exists at all.

Why not just train each candidate for five epochs? NAS-Bench-ASR reports that early-epoch rankings diverge from 40-epoch rankings among top models. Zero-Cost Proxies for Lightweight NAS reports that cutting epochs 64-fold yields only about 4× wall-clock savings. Because the 5-epoch numbers already exist in the table, we can plot top-decile Kendall tau against wall-clock cost for both approaches. If 5-epoch training dominates at comparable cost, we report that.

Why not just use a tuned standard model, as NAS-Bench-360 suggests? NAS-Bench-360 shows that tuned Wide ResNets often match NAS on non-vision tasks. We take that as a constraint, so the uniform Conformer is a first-class baseline. The Conformer NAS gain from 8.3% to 7.5% shows that heterogeneity can matter in speech. Our contribution is only meaningful if it reaches that kind of gain more cheaply.

Decisive Checks

The first check is the profile itself. Its value at initialization must correlate with its value after training on held-out models in at least two of the three families.

The second check is the central one. On the Conformer and TCN ground-truth sets, adding the measured profile to the analytic-plus-synflow feature set must raise top-decile Kendall tau under grouped cross-validation, where held-out architectures share no layer configuration with training architectures.

The third check is streaming. Measured lookahead must predict the latency-constrained CER ordering when candidates are filtered at 190 ms and 550 ms. These are the targets from the latency-controlled NAS paper, applied to our Conformer set run with chunked attention.

The fourth check is end-to-end search. Proxy-guided evolution must match DARTS with the Dynamic Search Schedule on AISHELL-1 CER at lower total GPU hours.

As a transfer stress test, we apply the proxy fitted on AISHELL-1 to about 60 encoder variants on SLUE-VoxPopuli named entity localization. SLUE Phase-2 found that this task depends on temporal alignment more than on transcription error. If the proxy is predictive anywhere outside ASR, it should be there.