When a paper claims architecture A beats architecture B, the claim is almost always made under one training recipe: one optimizer, learning rate, weight-decay value, augmentation pipeline, and epoch count. A long line of reproducibility work shows these claims often dissolve once the older model gets a modern or properly tuned recipe. Retuned CAML beats newer medical-coding models. Tuned GCN matches graph transformers. PointNet++ with a new recipe beats its successors. Yet the tools the field uses to rank architectures, including benchmarks such as NAS-Bench-201 and the cheap proxies built on them, still treat each architecture as having one true accuracy under one recipe.
This proposal asks a narrower and more answerable question. For a given set of architectures, which pairwise orderings are actually identified, meaning they hold across a reasonable family of training recipes? Which orderings are artifacts of the one recipe that happened to be chosen? We propose to measure this directly by retraining a structured sample of architectures under a designed grid of recipes and random seeds. We then split each pairwise accuracy gap into three parts: a part due to architecture, a part due to how architecture and recipe interact, and a part due to seed noise. From that measurement we build an adaptive evaluation procedure. It spends extra training runs only on the architecture pairs whose order is still uncertain, and its output is a partial ranking with calibrated confidence rather than a single total order.
Core Claim
The central hypothesis is about where ranking breaks down. Among the best architectures, the interaction between architecture and recipe is larger than seed noise, so a fixed-recipe benchmark confidently orders many top pairs that are not identified. Among weak architectures, the interaction term is small, and rankings are recipe-robust.
If this holds, two things follow. First, it explains the observation in Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization that early-versus-final rank correlation collapses in the top 1% of configurations. That collapse would be recipe mismatch rather than pure noise, a question the research file lists as open. Second, it implies that summary rank correlations such as Kendall's tau (a statistic measuring how often two rankings agree on the order of pairs) overstate how trustworthy a ranking is, because most of the pairs they count are easy ones.
The hypothesis is falsifiable. If the interaction term turns out to be at or below seed noise, even among top architectures, then a single fixed protocol plus enough seeds is adequate. We would report that as the result.
Position Relative to Prior Work
Three groups of work motivate the question.
The first is the confounding literature. Parameter Selection: Why We Should Pay More Attention to It shows that a grid search lifts CAML's Macro-F1 on MIMIC-III-50 from 0.532 to 0.635. Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark shows that feature normalization and tuning close the gap between message passing networks and graph transformers. PointNeXt shows that PointNet++ with a new recipe beats later architectures. These papers show that individual claims flip, one at a time. They do not estimate how often orderings flip across a whole space, or for which pairs.
The second group is the controlled-protocol studies. Pitfalls of Graph Neural Network Evaluation and Are GANs Created Equal? A Large-Scale Study give every model an equal tuning budget under a shared optimizer setup. NAS-Bench-201 fixes one recipe for 15,625 cells. Its authors acknowledge that some architectures may have been penalized by the single learning rate and weight decay, and we take that caveat as our starting point. We reuse its search space and treat its canonical recipe as one point in our recipe grid.
The third group supplies ranking methods that we audit rather than extend. Rethinking Performance Estimation in Neural Architecture Search finds a cheap training budget that preserves rank under one full recipe. Delta-NAS predicts pairwise accuracy differences between neighboring architectures. VKDNW scores untrained networks and argues for nDCG (normalized discounted cumulative gain, a ranking metric that weights correctness at the top of the list more heavily) over Kendall's tau. All three assume a recipe-fixed ground truth; we test whether that ground truth is stable. From VKDNW we borrow the emphasis on the top of the ranking. From VDA we borrow the principle that evaluation effort belongs where candidates disagree. VDA applies it to test instances of fixed models; we apply it to architecture pairs whose order is recipe-sensitive.
Two papers are the closest prior work. FBNetV3 jointly searches architecture and recipe. Its NARS recipe (the training recipe found by that joint search) raises ResNet50 from 76.1% to 79.2% on ImageNet, which is direct evidence of coupling between architecture and recipe. But FBNetV3 optimizes for one best pair inside a single block family and never measures ranking stability. AutoHAS searches hyperparameters and architecture together using shared weights, a setup that itself confounds architecture assessment. Tabular Benchmarks is the nearest dataset, since it crosses hyperparameters with MLP width and depth. Its analysis, however, targets hyperparameter-optimization algorithms rather than the question of which architecture orderings survive a recipe change. Best Practices for Scientific Research on NAS reports that training tricks contribute about 3% accuracy against less than 1% from the architecture. This is the magnitude argument for why our interaction term could plausibly be large.
Research Program
The core measurement is simple. For architectures $a$ and $b$, recipe $r$, and seed $s$, let $d_{ab}(r,s)$ be the test-accuracy difference $\text{acc}_a(r,s) - \text{acc}_b(r,s)$. We fit a random-effects decomposition:
$$d_{ab}(r,s) = \mu_{ab} + \gamma_{ab}(r) + \epsilon_{ab}(r,s).$$
Here $\mu_{ab}$ is the recipe-averaged gap, $\gamma_{ab}(r)$ is the recipe-specific deviation (the interaction), and $\epsilon_{ab}(r,s)$ is seed noise. We estimate the ratio of the interaction variance to the seed variance, overall and stratified by the pair's quality rank. A pair counts as identified if the sign of $d_{ab}$ is the same under at least a fraction $q$ of recipes, with seed uncertainty accounted for by a bootstrap. We report results for $q = 0.9$ and for a "peak" target, which compares each architecture's best recipe after averaging over seeds.
The work proceeds in three stages.
The first stage is a zero-compute pilot on the Klein–Hutter tabular benchmark. Its 62,208 configurations with four seeds already cross learning rate, batch size, dropout, and schedule with layer widths and activations. We treat the width, activation, and depth factors as the "architecture" and the rest as the "recipe," then run the decomposition. This tests whether the top-1% collapse reported there comes from the interaction term or from noise, before any GPU is spent.
The second stage is the main audit on the NAS-Bench-201 space for CIFAR-10 and ImageNet-16-120. We select about 200 architectures: the top 100 under the canonical recipe plus 100 sampled evenly across quality levels. Each is trained under a designed grid of roughly 16 recipes, with two seeds per recipe. The grid varies learning rate, weight decay, epoch count, warmup, and augmentation strength (standard crops versus cutout- or mixup-style regularization). The grid always contains the canonical NAS-Bench-201 recipe, so we can check our runs against the benchmark's published numbers. That comes to about 6,400 runs of small cell networks, which fits in a few thousand GPU hours.
The third stage is a smaller cross-family check on CIFAR-100. It covers eight backbones spanning residual CNNs, ConvNeXt-style CNNs, small ViTs, and MLP-mixers, trained under the same recipe grid with three seeds. The purpose is to see whether the within-space findings carry over to comparisons between families. Those are the comparisons that An Impartial Take to the CNN vs Transformer Robustness Contest found to depend on matching design choices.
From the full grid we define the reference partial order, then build an adaptive evaluator. The evaluator starts from the canonical-recipe ranking. It keeps a bootstrap posterior over the sign of each pair's gap among the current top candidates. It then assigns each new training run, as an (architecture, recipe) pair, to whichever candidate pair currently has the most uncertain sign. When the budget runs out, it outputs the pairs it can declare identified at a stated confidence. The grid runs are reused as a lookup table, so the evaluator can be tested at many budgets without retraining.
Baselines
The naive reference point is the canonical NAS-Bench-201 ranking: each architecture's accuracy under the benchmark's single recipe, at no extra cost. NAS-Bench-201 reports a best single architecture at 94.37% on CIFAR-10 and 47.31% on ImageNet-16-120. We measure what fraction of that ranking's confident top-pair orderings the reference partial order contradicts.
The second baseline spends the same extra compute as our adaptive evaluator, but only on more seeds under the fixed recipe. It isolates whether seed noise alone explains ranking errors.
The strongest competitor is equal-budget per-architecture tuning, the protocol of Pitfalls of Graph Neural Network Evaluation and Are GANs Created Equal?. Every architecture gets the same number of random draws from the same recipe space, and its best validation result is taken. This is the current field standard for fair comparison, and it could plausibly recover the peak target on its own. In Pitfalls, it made GCN the best model, with 99.4% relative accuracy and the best average rank. However, no reported result was found on Lacuna for its rank stability or pairwise-identification rate over a NAS search space. We will run it with the same recipe space, the same total number of training runs, and the same validation split as our evaluator.
We also include a ranking-predictor reference. It is the Delta-NAS pairwise-difference predictor, which reports Kendall's tau of 0.852 on NAS-Bench-101 in Delta-NAS, trained on canonical-recipe labels. We test how its tau against the reference order compares to its tau against canonical labels.
The ablation that isolates our key ingredient is the adaptive evaluator with uncertainty-driven allocation replaced by uniform allocation across the same candidate pairs.
We count our evaluator as beating a baseline under two conditions. First, at equal compute, it must match the reference top-10 set with higher nDCG. Second, its claims made at a stated confidence (for example, "A beats B with 90% confidence") must hold at close to that rate. The equal-budget tuning baseline makes no such claims at all.
Why not just…?
Why not just tune every architecture with an equal budget and report the result? This is the real threat. If equal-budget tuning already reproduces the peak target, our allocation adds nothing on that target. Equal-budget tuning still returns one number per architecture and says nothing about which top orderings would flip under a different draw of tuning trials. We will test that directly by rerunning the tuning with different random draws and measuring how often the top-10 orderings change. If they rarely change, we will say the simple method suffices for the peak target. The contribution then reduces to the decomposition and to the robust ($q$-fraction) target, which the simple method does not address.
Why not just add more seeds under the benchmark recipe? Seeds only estimate the noise term $\epsilon$. They cannot reveal the interaction term $\gamma$, because it is invisible whenever the recipe is held fixed. The seed-matched baseline and the variance ratio settle this. A ratio well below one means more seeds is the right fix.
Why not just compute Kendall's tau between rankings under different recipes? We compute it, but it is dominated by easy pairs among weak architectures. It therefore mixes recipe-robust and recipe-fragile regions, which is the concern behind VKDNW's argument for nDCG and the Klein–Hutter top-1% finding. The decomposition by quality level is what reveals where the fragility sits.
Why not just use FBNetV3-style joint architecture and recipe search? Joint search answers "what is the best pair," which is a different question from "which comparisons are trustworthy." Its predictor would also have to be trained on the very grid we are building. FBNetV3 also reports no stability measurement that we could compare against.
Decisive Checks
The project succeeds or fails on three measurements.
The first is whether, among the NAS-Bench-201 top 100, the interaction-to-seed variance ratio exceeds one while staying well below one for the evenly sampled architectures. The same pattern should appear first in the free tabular pilot.
The second is how many canonical-recipe top-pair orderings are unidentified under the $q = 0.9$ target, and whether the robust and peak targets disagree with each other.
The third is whether the adaptive evaluator reaches a given top-10 nDCG with fewer training runs than equal-budget tuning, with calibrated confidence on the pairs it declares identified. We will also check whether the cross-family CIFAR-100 set shows the same concentration of fragility among the strongest models.
A negative outcome on the first check is a publishable result in its own right. It would be evidence that single-recipe benchmarks are trustworthy even at the top.
References
- Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
- Parameter Selection: Why We Should Pay More Attention to It
- Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark
- PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
- Pitfalls of Graph Neural Network Evaluation
- Are GANs Created Equal? A Large-Scale Study
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Rethinking Performance Estimation in Neural Architecture Search
- Delta-NAS: Difference of Architecture Encoding for Predictor-Based Evolutionary Neural Architecture Search
- Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights
- VDA, a Method of Choosing a Better Algorithm with Fewer Validations
- FBNetV3: Joint Architecture-Recipe Search Using Predictor Pretraining
- AutoHAS: Efficient Hyperparameter and Architecture Search
- Best Practices for Scientific Research on Neural Architecture Search
- An Impartial Take to the CNN vs Transformer Robustness Contest