Speech systems almost always normalize their input features before a network sees them. A common example is cepstral mean and variance normalization (CMVN), which subtracts each recording's average spectrum and rescales it so that microphone and speaker differences shrink. This helps speech recognition. It also deletes information other tasks need. Speaker identity, prosody (the rhythm and pitch of speech), and the channel cues that let a separator pull one voice out of a mixture are all partly erased. Separately, neural architecture search (NAS), the automated design of network structure, is expensive because each candidate network has to be trained to be scored. The cheap shortcuts, called "zero-cost proxies," score a randomly initialized network from one forward or backward pass. These proxies were developed almost entirely on images, and they treat the input pipeline as fixed.
This note proposes to search jointly over the normalization choice and the architecture. The core contribution is a zero-cost score that depends on the task. Given a candidate normalization, a candidate cell, and a target task, it measures at random initialization how much of the target task's label structure survives in the network's features. The hypothesis is that the best normalization depends on the task and also on the architecture. Phone recognition and speaker identification on the same audio should then rank the same normalization-plus-architecture pairs differently.
Existing proxies cannot see this difference. Synflow is computed on an all-ones input, so input normalization cannot affect it, and it returns the same ranking for every task. If the hypothesis holds, a cheap task-aware score should recover rankings that existing proxies are blind to, at a small fraction of the cost of the short training runs used today.
Core Claim
Normalization-induced information loss is a property of a normalization, an architecture, and a task taken together. It is not a fixed preprocessing decision. The claim has two parts.
The first is empirical. Across a sample of speech cells, the ranking of (normalization, cell) pairs under phone error rate differs substantially from the ranking under speaker-identification accuracy. Some of that difference should come from interaction: certain cells recover what a given normalization removes, and others do not.
The second is methodological. A label-alignment score computed at initialization ranks these pairs better than existing zero-cost proxies. It does so after controlling for parameter count, and it costs orders of magnitude less than five-epoch training.
Where This Sits
Several papers motivate the gap by documenting the trade-off directly:
- Stolcke et al. show that vocal-tract length normalization, tuned for recognition, removes the anatomical cues speaker recognition relies on.
-
Schatz et al. show that making features fully invariant to phonetic variation also degrades speaker discrimination. - Mandarin tone recognition using affine-invariant prosodic features shows that normalizing at the syllable level erases the pitch and intensity cues that carry lexical tone.
-
Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation reports that feature normalization lowers separation quality by up to 1.9 dB of scale-invariant SNR improvement (SI-SNRi). It also finds that raw log-mel filterbanks (13.3 dB) beat normalized speaker embeddings (12.4 dB) with Conv-TasNet on LibriSpeech.
All of these studies evaluate hand-picked, fixed architectures. None asks whether the size of the loss depends on the network that follows the normalization.
On the search side, NAS-Bench-ASR trained 8,242 speech cells on TIMIT under a single fixed input pipeline. It found that the correlation between validation and test error scatters among the top 1,000 models. That finding warns that shortcut scores are fragile in speech.
Three papers supply the proxy methodology:
- Zero-Cost Proxies for Lightweight NAS defines the standard proxies (synflow, snip, grad_norm, fisher, jacob_cov).
- NAS-Bench-Suite-Zero standardizes 13 proxies over 28 tasks. None of those tasks is speech recognition or separation.
- EZNAS evolves proxies automatically.
Two of these papers also constrain the design. NAS-Bench-Suite-Zero documents a strong bias toward large models, and EZNAS's best evolved proxy turns out to be a nonlinear parameter counter. Any new proxy therefore has to beat parameter count and FLOPs, not just correlate with accuracy.
The work closest to this proposal is BN-NAS. It uses batch-normalization scale parameters as an architecture-importance signal for convolutional vision supernets. Here, by contrast, normalization is treated as a searched source of information loss whose effect depends on the task, and it is applied to speech inputs. RNAS constrains feature distortion during differentiable search. It treats feature divergence as harmful by definition. This proposal assumes the opposite: the variance a normalization removes may be exactly the signal a second task needs.
Research Program
Search space. The joint space crosses a sample of NAS-Bench-ASR cells, inside the benchmark's own macro-structure, with four input normalizations:
- no normalization (raw log-mel filterbanks);
- utterance-level CMVN;
- causal running mean subtraction, the streaming normalization that Maas et al. and King et al. found blurs boundaries between speakers;
- short-window mean and variance normalization at roughly syllable scale, the setting that destroys tone cues.
Tasks. Both tasks use TIMIT, so the audio is held fixed and only the target labels change. The first is frame-level phone recognition, scored by phone error rate (PER). The second is speaker identification over TIMIT's speakers. This is the cleanest available contrast, because the two tasks need opposite invariances. A stretch task is target speaker separation on LibriSpeech mixtures, where normalization has a known cost.
Ground truth. About 250 stratified cells are crossed with the 4 normalizations and the 2 tasks, giving 2,000 configurations. Each is trained with two seeds at the NAS-Bench-ASR 40-epoch budget, about 4,000 runs in total. NAS-Bench-ASR completed more than 74,000 runs of comparable size, so this budget is feasible. As a sanity anchor, the reproduced setting closest to the benchmark's pipeline should correlate with the published 40-epoch PER. If it does not, the reproduction is broken.
Interaction analysis. Variance in final performance is decomposed into a normalization effect, a cell effect, and a normalization-by-cell interaction, separately for each task. The decomposition also counts how often the best normalization for a given cell differs from the best normalization averaged over all cells.
Proposed proxy. Take a candidate network at random initialization. On Random Weights and Unsupervised Feature Learning supports the premise that random networks already carry usable structure. Pass one labeled batch of normalized frames through it. Score the network by centered kernel alignment between the feature similarity matrix and the target-label similarity matrix:
$$S(n, c, t) = \mathrm{CKA}\big(K_{f_c(n(x))},\, K_{y_t}\big)$$
The symbols are:
- $n$ is the normalization and $c$ is the cell;
- $f_c(n(x))$ are the cell's last-block features on normalized inputs $x$;
- $y_t$ are the labels for task $t$;
- each $K$ is a Gram matrix, i.e. the matrix of pairwise similarities.
The score is averaged over three random initializations. It needs labels but no training. Because CKA is computed on similarity matrices, it is insensitive to feature width, which partly addresses the large-model bias.
Baselines
Naive reference points. The first naive baseline keeps one normalization, utterance CMVN, and ranks cells by parameter count. The second ranks cells by FLOPs. Given the size bias documented in NAS-Bench-Suite-Zero, either could already be strong. The proposed score must beat both. It must also keep a significant partial rank correlation with final performance after parameter count is controlled for.
Existing zero-cost proxies. These are synflow, snip, grad_norm, fisher, jacob_cov, and the three-proxy voting ensemble from Zero-Cost Proxies for Lightweight NAS. All are computed on the same normalized batch, the same initializations, and the same cells. The ensemble reports Spearman ρ = 0.82 on NAS-Bench-201, and EZNAS reports Kendall τ = 0.65 on NAS-Bench-201 CIFAR-10. Both numbers are on images. No reported result was found on Lacuna for zero-cost proxies on NAS-Bench-ASR, so this project's own runs supply the speech number. To beat these proxies, the proposed score must achieve higher Spearman correlation on the joint (normalization, cell) space for both tasks. It must also achieve lower top-10 regret, measured as the gap between the selected configuration's final metric and the true best.
Strongest competitor: short training. This is five-epoch training, the low-budget tier already present in NAS-Bench-ASR. It is the budgeted estimation that Rethinking Performance Estimation in Neural Architecture Search shows can reach about 1,000x speedups. It is run on the same configurations and hardware, and wall-clock time is reported. No reported result was found on Lacuna for the five-epoch-to-forty-epoch rank correlation on NAS-Bench-ASR, so it is measured here. Ranking correlation is not the bar for beating it, since a proxy may lose to it there. The bar is matching its top-10 regret within one seed's standard deviation at less than 1% of its wall-clock cost. The comparison also checks whether it is fooled more often than the proposed score among top models, where NAS-Bench-ASR found validation and test error diverge.
Key ablation: normalization picked independently of the architecture. This baseline isolates the proposal's main ingredient. For each task, it fully trains one reference cell under each normalization, fixes the winning normalization, and then ranks cells by the proposed score under that normalization only. If this matches joint selection, the interaction claim fails. A second ablation removes task conditioning by scoring alignment with a fixed proxy label set, using phone labels for both tasks. It tests whether task dependence is what gives the score its value.
Why not just…?
Why not just drop normalization and feed raw filterbanks? The separation results in Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings make this a real competitor, and it is one of the four searched options. The trade-off literature, however, says normalization still helps recognition (Stolcke et al.). If raw filterbanks win both tasks for nearly every cell, the interaction analysis will show it, and the conclusion will simply be to stop normalizing.
Why not just train each candidate for five epochs? This is the most serious threat to the proxy's usefulness. Short training does see the effect of normalization, unlike synflow. The two arguments against relying on it both come from the file. First, Zero-Cost Proxies for Lightweight NAS reports that a 64x FLOP reduction yields only about 4x wall-clock savings. Second, NAS-Bench-ASR found that rank stability breaks down among top models. The direct comparison of regret against cost settles the question. If five-epoch training wins clearly on regret, the proxy is useful only for cheap pre-filtering.
Why not just accept that zero-cost scores are disguised parameter counts? EZNAS found exactly this for its best evolved proxy, and it may hold for this score too. Partial correlation against parameter count and FLOPs is therefore a primary metric, not an afterthought. A useful sign is the across-task check. For a fixed cell, size is constant, so any change in the score's ranking between phone and speaker tasks cannot come from size.
Why not just choose the normalization once per task with a reference model? If normalization and architecture barely interact, this simple two-stage procedure is enough, and joint search adds nothing. The first ablation tests exactly this. The result is reported whichever way it comes out.
Why not just read middle layers of a pretrained self-supervised model? Phone and speaker spatial organization in self-supervised speech representations finds middle-layer "sweet spots" and argues that the pretraining objective shapes representations more than the architecture does. That is a different regime, frozen foundation models rather than compact trained-from-scratch models for on-device use, which is where NAS-Bench-ASR's latency measurements matter. This proposal does not compete there. Its finding would, however, temper expectations if the cell effect turns out small relative to the normalization effect.
Decisive Checks
The project succeeds if three things hold. The normalization-by-cell interaction explains a non-trivial share of variance, and the best normalization for a cell differs from the best overall normalization for a meaningful fraction of cells, in at least one task. The proposed score beats every existing zero-cost proxy and parameter count on joint-space Spearman correlation and top-10 regret, for both phone and speaker tasks. And it approaches five-epoch training's regret at a tiny fraction of its cost.
The project fails cleanly in three cases. The independent-selection ablation matches joint selection. Or the score loses its signal once parameter count is partialled out. Or synflow, which cannot see normalization at all, ranks configurations as well as the proposed score. Any of these outcomes is publishable, because it would tell speech practitioners whether normalization can safely be chosen independently of the architecture.
References
- MLLR transforms as features in speaker recognition
- Evaluating speech features with the minimal-pair ABX task: analysis of the classical MFC/PLP pipeline
- Mandarin tone recognition using affine-invariant prosodic features and tone posteriorgram
- Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation
- NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition
- Zero-Cost Proxies for Lightweight NAS
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- BN-NAS: Neural Architecture Search with Batch Normalization
- Robust Network Architecture Search via Feature Distortion Restraining
- Anchored Speech Detection
- Robust Speech Recognition via Anchor Word Representations
- On Random Weights and Unsupervised Feature Learning
- Rethinking Performance Estimation in Neural Architecture Search
- Phone and Speaker Spatial Organization in Self-Supervised Speech Representations