Is the Proxy Transfer Gap Partly a Training-Recipe Artifact?

Automated architecture search almost never trains candidates on the task we care about. It ranks them on a cheap stand-in, called a proxy task: a smaller dataset such as CIFAR-10, a shorter training run, or a score computed at initialization. It then assumes the winners stay winners on the real target, such as ImageNet. That assumption often fails, especially among the best candidates. The standard benchmarks that document this failure share one design choice: every architecture is trained with a single fixed recipe, meaning the same optimizer, learning rate, weight decay, augmentation and epoch count, on both the proxy and the target. A practitioner deploying an architecture would tune its recipe. So the "true" target ranking these benchmarks report is itself defined by a recipe nobody would actually use.

This note proposes to measure how much of the proxy-to-target ranking failure comes from recipe choice and how much is a real property of the architectures. It then tests a simple correction. The proxy recipe would be chosen to maximize agreement with the target ranking on a small calibration set of architectures, rather than to maximize proxy accuracy. If the recipe-induced part is large, the field has been misattributing a protocol problem to architecture scaling. It would also mean the impressive correlations reported for cheap proxies are partly correlations with one arbitrary recipe. If the recipe-induced part is small, that is equally useful: it would confirm the gap is intrinsic and settle a confound in the benchmark literature.

Core Claim

The claim has two parts. The first is about measurement. The ranking gap between proxy and target should be measured against a tuned target ranking, in which each architecture gets its best recipe from a fixed search budget. It should not be measured against a uniform-recipe ranking. Measured that way, the gap splits into a recipe-mismatch component and an intrinsic component, and the split can be estimated. The second part is about method. A proxy protocol calibrated for rank agreement on roughly fifty architectures removes much of the recipe-mismatch component on held-out architectures. It also does better than the same calibration done for proxy accuracy, which is what practitioners implicitly do today.

Prior Work

The motivation comes from the controlled-evaluation literature, which shows that recipe choices routinely dominate apparent architecture effects.

All of these studies vary recipes for a fixed task. None asks how recipe choice affects the agreement between a proxy ranking and a target ranking, which is the question here.

The constraint comes from the uniform-recipe benchmarks. NAS-Bench-201 and NATS-Bench trained 15,625 cell topologies, plus 32,768 width configurations in NATS-Bench, under one recipe on CIFAR-10, CIFAR-100 and ImageNet-16-120. NATS-Bench reports that rank agreement collapses among the top 1% of candidates. Its authors concede that the single recipe may keep some topologies below their peak, and the Ludwig Benchmarking Toolkit makes the same point for text models. Neither benchmark separates recipe effects from architecture effects in the transfer gap. We reuse their search spaces and APIs as the starting point.

The most direct evidence that the proxy protocol matters is Less is More. Changing only the proxy data, to a 25% subset chosen by autoencoder loss, moves DARTS-V2 on CIFAR-100 from 53.75% to 75.20%. Accelerating NAS via Proxy Data shows a similar effect with entropy-tail subsets. Both vary data selection only and judge success by the accuracy of the single architecture found. We instead vary optimization and regularization, and judge success by rank agreement across many architectures.

NASTransfer documents that proxy-searched architectures often fail to beat random sampling on ImageNet1K and ImageNet22K, with a particularly bad "CIFAR-100 trap". It does not ablate the proxy recipe.

Two lines of work treat the gap as structural rather than procedural. AdaptNAS treats it as domain shift and bridges it with a rotation-prediction task and adversarial alignment. EAT-NAS argues that cell structure must deform across resolutions. Our proposal is complementary, because it leaves architectures and losses untouched and changes only recipe hyperparameters. If the intrinsic component turns out to dominate, these works are vindicated.

Finally, Zero-Cost Proxies for Lightweight NAS scores architectures at initialization. An ensemble of three such scores (Synflow, jacob_cov and SNIP) reaches Spearman 0.82 against the uniform-recipe NAS-Bench-201 accuracies. A score computed before any training cannot respond to the recipe. So its correlation with a tuned target is an open and testable question, and this proposal answers it.

Research Program

Search spaces and datasets. The main testbed is the NATS-Bench topology space, with CIFAR-10 and CIFAR-100 as proxies and ImageNet-16-120 as the target. We draw about 300 architectures. The draw over-samples the top decile of the published uniform-recipe accuracies, so that rank agreement among the top candidates, where the gap is known to live, can be estimated with usable precision.

Recipe grid. Each architecture is trained on proxy and target under a grid of about twelve recipes. The grid varies peak learning rate, weight decay, epoch budget (including short budgets of the kind EcoNAS-style proxies use), augmentation strength and label smoothing. It includes the original NAS-Bench-201 recipe as one cell. Three seeds per cell let us separate recipe effects from seed noise. This yields a tensor of accuracies indexed by architecture, recipe, dataset and seed.

Defining the decomposition. Let $R_P(r)$ be the proxy ranking under recipe $r$. Let $R_T^{u}$ be the target ranking under the uniform recipe, and $R_T^{*}$ the tuned target ranking, which takes each architecture's best recipe on a held-out validation split. We report three quantities.

  • The benchmark-reported gap, $1-\tau(R_P(u), R_T^{u})$, where $\tau$ is Kendall rank correlation.
  • The practitioner-relevant gap, $1-\tau(R_P(u), R_T^{*})$.
  • The best achievable gap, $1-\min_r$ of the same quantity.

The difference between the uniform-recipe gap and the best-recipe gap is the recipe-induced component. What remains after recipe optimization, compared with the seed-noise ceiling estimated from agreement between seeds, is the intrinsic component. A two-way analysis of variance on architecture and recipe, of the kind used by One Model, Many Behaviors, quantifies how strongly architectures and recipes interact. If the interaction is large, no single recipe can be fair to all architectures.

The calibrated proxy protocol. We choose the proxy recipe that maximizes Kendall agreement with $R_T^{*}$ on a calibration set of $k$ architectures, varying $k$ from 10 to 100. We then evaluate on the remaining held-out architectures. All reported costs include the calibration training on the target.

Transfer of the calibrated recipe. To test whether calibration is a reusable property of the protocol rather than a fit to particular architectures, we take a recipe calibrated on the topology space and apply it unchanged to the NATS-Bench width space.

Full-resolution check. In a small confirmatory study, about 20 cells from the DARTS space used by NASTransfer are trained at full ImageNet1K resolution. A few recipes per cell suffice to check whether the direction of the effect survives the resolution jump that EAT-NAS identifies.

Baselines

The naive reference point is the uniform-recipe proxy itself. Candidates are ranked by CIFAR-10 accuracy under the NAS-Bench-201 recipe, exactly as the benchmark tables provide. Every other method must beat it on agreement with the tuned target, both globally and among the top 5% and top 1% of architectures. Success also includes lower regret, defined as the tuned target accuracy of the best architecture available minus that of the architecture each method selects.

The strongest existing cheap ranker is the zero-cost ensemble from Zero-Cost Proxies for Lightweight NAS. It reports Spearman 0.82 on NAS-Bench-201 against uniform-recipe accuracy, compared with 0.61 for the reduced-training proxy EcoNAS. We recompute it on our exact architecture sample against both $R_T^{u}$ and $R_T^{*}$. It is the strongest competitor because it is essentially free and already correlates well. The calibrated protocol beats it only if it achieves higher agreement with $R_T^{*}$, and lower regret, over the top decile of held-out architectures. Overall agreement on the full set does not count, since it is dominated by easy-to-separate weak architectures.

The strongest data-side competitor is searching on a subset of the target data. We implement the entropy-tail selection of Accelerating NAS via Proxy Data on ImageNet-16-120, sized so its total GPU-hours equal the calibrated protocol including calibration cost. There is no reported result on Lacuna for the rank correlation of entropy-tail or autoencoder-sampled proxy data with tuned target accuracy. That paper and Less is More report only the accuracy of the single architecture found.

Uniform random sampling, which NASTransfer finds hard to beat, is reported as the regret floor.

The key ablation isolates the calibration objective. It calibrates the proxy recipe by maximizing mean proxy accuracy, which is what tuning a CIFAR-10 pipeline normally means, using the same grid, the same $k$ and the same budget. If rank-agreement calibration does not beat accuracy calibration, the proposal's central ingredient adds nothing, and any gain is just "a better-tuned proxy".

All methods share the same architecture splits, seeds and target ground truth. Grid-based methods share the same grid.

Why not just…?

Why not just use zero-cost proxies? They may already be good enough, and that is a real outcome. Their reported 0.82 is measured against uniform-recipe accuracy, however, and a score computed at initialization cannot track which recipe suits which architecture. If the tuned target ranking differs substantially from the uniform one, their agreement with the tuned target should drop. That drop is what the first decisive check below measures. If it does not drop, we report that zero-cost proxies are robust to recipe choice, which is itself a new and useful result.

Why not just search on a small subset of the target data? This is the most serious threat, because it skips the transfer problem altogether. Na et al. and AdaptNAS both search on ImageNet subsets cheaply. Two things work against it. Subset selection is itself a protocol choice, with the same uncalibrated-recipe problem. And very small subsets may distort rankings among the top candidates, as the CIFAR-100 trap suggests low sample density per class can. The equal-cost comparison settles the question. If a target subset matches or beats the calibrated proxy at equal GPU-hours, we recommend calibrating the recipe applied to the target subset instead, and the decomposition result still stands.

Why not just run random search? NASTransfer shows random sampling is competitive. It still requires training many candidates on the target, and its regret depends entirely on how good the search space is. We report it as the regret floor. The proposal is mainly about how faithfully proxies rank candidates, and random search does not answer that.

Why not just train the proxy longer? Longer training is one axis of the recipe grid, so this alternative is tested directly rather than argued against. If the longest-budget uniform recipe captures most of the recipe-induced gain, the conclusion is a simple one, "train proxies longer", and we will say so.

Decisive Checks

Four results decide the outcome.

  1. Do the two target rankings differ? We compute Kendall agreement between $R_T^{u}$ and $R_T^{*}$ among the top decile. If it is near the seed-noise ceiling, recipes do not reshuffle target rankings. The recipe-artifact hypothesis is then false and the study reduces to a clean negative result.
  2. Does recipe-matched transfer close the gap? We test whether the best-recipe gap is much smaller than the uniform-recipe gap among the top 1% of architectures, which is where NATS-Bench reports the breakdown.
  3. Does calibration generalize? A recipe calibrated on 50 architectures should beat both the uniform recipe and accuracy-calibrated recipes on held-out architectures. It should also keep that advantage when moved to the width space.
  4. Does it survive full resolution? The ImageNet1K check should preserve the direction of the effect, even though 20 cells only allow a coarse estimate.

A positive answer on the first two checks with a negative answer on the third is still informative. It would mean the gap is recipe-induced but architecture-specific, which argues for per-architecture tuning rather than any single shared proxy protocol.