Neural architecture search, the automated design of network wiring, has largely solved its coarse problem. Cheap scores computed on untrained networks, and short training runs, reliably separate bad designs from good ones. What remains expensive is the last step: deciding which of the few hundred very good candidates is actually best. Those candidates differ by fractions of a percent. Cheap scores cannot tell them apart, and short training runs scramble their order. This note proposes a fix taken from adversarial curriculum generation, in which a "teacher" chooses training tasks to expose a learner's weaknesses. During the short training runs used to rank finalist architectures, an adaptive teacher picks training data and augmentation strength. The teacher is rewarded only for data that some finalist can learn and others cannot.
The idea in two sentences: short training runs rank top architectures poorly because most of the budget goes to easy examples that every good architecture learns equally well, plus impossible examples that none learns. A teacher rewarded by regret, meaning the gap between the best finalist and a given finalist on an example, steers the fixed budget toward the examples that actually separate architectures, so the same GPU hours should produce a ranking closer to the full-training ranking.
Core Claim
At a fixed per-candidate training budget, a regret-rewarded data teacher shared across a population of top-tier candidates should raise rank agreement with full-training accuracy among the top 10% of NAS-Bench-201. Rank agreement is measured by Kendall's tau, the fraction of correctly ordered pairs minus the fraction of incorrectly ordered pairs. We expect this gain over uniform data, over static proxy data, and over a teacher rewarded by single-model difficulty. The gain should show up as lower regret of the finally selected architecture. The regret term is the essential ingredient. It is what filters out impossible and trivially easy data automatically, and this is exactly the property that made minimax regret work in environment design.
This combination is new relative to the closest work. Accelerating NAS via Proxy Data selects easy and hard tails of the data once, offline, using a frozen ResNet-20. Its data never adapts to the candidates being compared. CLOSE builds a curriculum over the supernet's internal weight-sharing degree on a hand-crafted schedule, and it feeds uniform batches. GA-NAS is adversarial, but its generator proposes architectures, not data. Its evaluation data stay static. Info-Coevolution and active-iNAS co-adapt data selection with a model, but they target labeling cost with standard acquisition scores rather than architecture comparison. None of these uses a cross-candidate regret signal to allocate training data.
Why Regret, and What It Borrows
The mechanism comes from PAIRED. There, an environment generator is rewarded by the antagonist's score minus the protagonist's score. Impossible tasks therefore earn zero reward, and so do trivial ones, which kept difficulty inside the learnable range where pure minimax adversaries made tasks unsolvable. PAIRED reports 40% success on a labyrinth versus 10% for minimax and 0% for domain randomization.
We transfer this calibration property and drop the two-agent RL setting. The finalists form a population. For a data cell $c$ (a cluster of training images paired with an augmentation-strength level) and candidate $a$, define the reward as
$$r(c,a) = \max_{a'} p_{a'}(c) - p_a(c),$$
where $p_a(c)$ is candidate $a$'s recent training-time learning progress on cell $c$, meaning the drop in its loss over a short window. The maximum runs over the current population. The teacher is a non-stationary bandit, a sampler that keeps updating which arms pay off as conditions change. It allocates each candidate's batches across cells in proportion to smoothed regret, with an exploration floor.
Augmentation strength is the "generation" knob, as in parameterized environment design. Distortions too severe for any finalist earn no regret and fade out. Distortions every finalist absorbs fade out as well. The same calibration concern appears in IFDecorator, which filters for intermediate success rates.
One caution comes directly from that line of work. PAIRED reports that adversarial dynamics can cycle and fail to converge. We sidestep most of this because the "players" are fixed candidates with short budgets, not co-adapting policies. We still track allocation entropy across training for oscillation.
Research Program
We will build the teacher as a lightweight layer around standard CIFAR training. It needs per-example loss logging, k-means clusters over features from a randomly initialized network to avoid importing a pretrained model's bias, and four augmentation levels.
The candidate pool is a stratified sample of 300 architectures from the top 10% of NAS-Bench-201 by 200-epoch CIFAR-10 accuracy, plus 100 from the top 30% as a control. Each candidate is trained under budgets of 4, 12, and 25 epochs on CIFAR-10, CIFAR-100, and ImageNet-16-120. Ground truth is the benchmark's 200-epoch accuracy from NAS-Bench-201.
We will also study a structural question: does curriculum training change which architecture is truly better at short budgets, or does it only reveal the true ordering earlier? Comparing against full-training ground truth settles this directly. A teacher that biases toward an artificial task would show high within-proxy consistency but low tau against ground truth.
Baselines
The naive reference is uniform data at the same epoch budget, which is what budgeted estimation in Rethinking Performance Estimation in NAS effectively does. That paper reduced evolutionary search to 0.33 GPU days at 2.66% CIFAR-10 error. We will also apply its finding that epochs matter most while image size can be cut, and give every method the same reduced-resolution setting.
The strongest data-centric competitor is static bimodal proxy data from Accelerating NAS via Proxy Data, which completed DARTS search on CIFAR-10 in 40 minutes using as few as 1.5K images. We run it with its entropy-tail selection at matched GPU hours, both by subsetting and by reweighting.
The strongest training-free competitor is the EZNAS-A zero-cost score from EZNAS. It reports Kendall's tau of 0.65 on NAS-Bench-201 CIFAR-10 over the full space, against 0.57 for NASWOT and Synflow. The weight-sharing reference is CLOSE, which reports 0.6693, against 0.6122 for K-shot NAS. Both figures are full-space correlations. We found no reported result on Lacuna for Kendall's tau restricted to the NAS-Bench-201 top 10%, so we will compute it ourselves for every method. EZNAS's own admission that zero-cost proxies fail to separate the top 10% predicts their score there.
We include two ablations to isolate the key ingredient. The first is a single-model hardness teacher that allocates by each candidate's own loss without the cross-population maximum. The second is a population-variance teacher that allocates by disagreement across candidates without the regret asymmetry. Every method gets the same per-candidate GPU hours (teacher overhead included), the same seeds, and a tuning budget of 20 configurations on a held-out set of 50 architectures.
We count it as beating the baselines if top-10% tau exceeds the best baseline by at least 0.05 with non-overlapping seed intervals at two of three budgets. Selected-architecture regret against the benchmark optima (94.37% CIFAR-10, 73.51% CIFAR-100, 47.31% ImageNet-16-120) must also fall.
Why not just…?
Why not just train the finalists longer with successive halving? Spending more epochs on survivors is the obvious cheaper move. We will plot top-10% tau against total GPU hours for both methods. If successive halving reaches the same tau at equal hours, the curriculum adds nothing. The proposal's claim is a better hours-to-tau curve, not a better endpoint. The curriculum also composes with halving, and we test that combination.
Why not just weight examples by where candidates disagree? This is the real threat. Cross-population variance already captures much of what regret captures, so it is run as a full ablation rather than dismissed. The difference shows up in one specific case. Variance rewards examples where outputs differ for idiosyncratic or noisy reasons, including mislabeled images on which candidates fail inconsistently. Regret requires that some candidate actually makes progress on the example. We will inject 10% label noise. If variance and regret tie both with and without noise, we will report that the simpler rule suffices and drop the bandit.
Why not just use zero-cost proxies or a learned predictor for the finalists? EZNAS states that no zero-cost proxy reliably separates the top 10%. Graph predictors such as FR-NAS improve tau by 3–16% in small-sample regimes, but they learn from labels produced by exactly the short training runs we are improving. Our experiments train FR-NAS on uniform-budget labels and on curriculum-budget labels to see whether the gain propagates.
Proposed Experiments
The decisive check is top-10% Kendall's tau against 200-epoch ground truth on all three NAS-Bench-201 datasets at the three budgets, for every baseline and ablation, with five seeds. Seed-to-seed tau of each proxy serves as a noise ceiling.
The second check is end-to-end. We run regularized evolution, the guided search that NAS-Bench-101 found matched random search's final result about five times faster, and replace its final ranking stage with each method. We then report selected-architecture regret at equal total GPU hours.
The third check tests the "zero regret for impossible data" property. We add a cell of fully scrambled images and confirm that the regret teacher assigns it near-zero mass, while the single-model hardness teacher over-samples it.
Finally, as a transfer probe, we apply the method unchanged to two NAS-Bench-360 tasks that Agentic NAS used, including the Spherical task where it reduced error from 45.08% to 38.71%. That paper also reported validation-to-test overfitting on DeepSea, so this probe asks whether regret curricula worsen or reduce validation-test divergence.
References
- Accelerating Neural Architecture Search via Proxy Data
- CLOSE: Curriculum Learning on the Sharing Extent Towards Better One-Shot NAS
- Generative Adversarial Neural Architecture Search
- Info-Coevolution: An Efficient Framework for Data Model Coevolution
- Deep Active Learning with a Neural Architecture Search
- Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
- IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Rethinking Performance Estimation in Neural Architecture Search
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- FR-NAS: Forward-and-Reverse Graph Predictor for Efficient Neural Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Agentic Neural Architecture Search