When a language model writes neural-network code in a loop (propose an architecture, train it, read the result, propose again), almost all of the cost goes to training each candidate from scratch. The obvious shortcut is to train only some candidates and predict the accuracy of the rest with a cheap learned predictor, called a surrogate. That shortcut has a known failure mode. An optimizer that maximizes a learned predictor will find the places where the predictor is wrong and drift toward them. Chemists ran into this years ago. Their fix was the applicability domain: a boundary drawn around a model's training data, outside of which its predictions are not trusted.
We propose bringing that fix into open-ended architecture search. Each piece of model code the language model writes is traced into an operator graph and summarized as a descriptor vector. An applicability-domain test, adapted from cheminformatics, then decides for each new candidate whether the surrogate's predicted accuracy can stand in for real training, or whether the candidate is far enough from everything already trained that it must be trained for real. In two sentences: the boundary does not restrict what the code-writing model may propose. It only decides which proposals can be scored cheaply, and it widens automatically as more candidates are trained.
Core Claim
The claim has two parts. First, in architecture search driven by a fitness predictor, the candidates the optimizer selects carry systematically larger predictor errors than random candidates. Second, a distance-based applicability-domain gate catches those errors well enough that most real training runs can be replaced by predictions, with no loss in the final architecture's true accuracy. Neither part has been shown for architecture search.
The motivating evidence comes from molecular design. Graph Machine Learning for Design of High-Octane Fuels showed that optimizers exploit a surrogate's blind spots and converge on artifacts. That paper responded by putting one-class SVMs and density estimators inside the loop to penalize drifting candidates. We reuse its detectors but change their role: they route each candidate to an evaluator rather than steering the search.
The method we reuse most directly is Intriguing Usage of Applicability Domain: Lessons from Cheminformatics Applied to Adversarial Learning (BAARD). BAARD stacks three checks:
- a global statistical test of distance from the training data,
- a local nearest-neighbour similarity test,
- a "decidability" test of whether the model's output agrees with its neighbours' outputs.
BAARD applies these checks to the inputs of a fixed network. We apply them to the architectures themselves, and to the surrogate that scores those architectures.
What Exists and What Is Missing
Surrogate predictors for architecture search are mature but have no notion of their own domain. NAS-Bench-301 fits graph networks and XGBoost ensembles over the DARTS and FBNet spaces and reports a sparse Kendall rank correlation of about 0.8. How Powerful are Performance Predictors in Neural Architecture Search? benchmarks 31 predictors and builds OMNI, a hybrid of zero-cost proxies, learning curves and a model-based meta-predictor, reporting rank correlation up to 30% higher than any single predictor. Both papers measure correlation on held-out samples from the same space. Neither measures error on the candidates an optimizer actually picks, which is where exploitation happens. Both also assume a fixed, enumerable search space.
The open-ended setting is where this matters most. Agentic Neural Architecture Search (AgentNAS) has a language-model team write a seed model in a sandbox, decompose it into slots holding 4–8 alternative modules each, and run regularized evolution over those slots. It trains every candidate from scratch on a single GPU. That cost pushes it toward small models and caps how many candidates it can try. Its authors also report that on DeepSEA, search gains on validation did not carry over to test.
Code-refinement loops for other artifacts likewise rely only on after-the-fact execution feedback:
- CAAFE uses execution feedback for feature-engineering code.
- POLO uses it for C/C++ performance rewrites.
- A2DEPT uses it for evolved solver programs.
In every case, each candidate is fully executed before anything is known about it. We take the AgentNAS-style loop as the setting, the cheminformatics domain tools as the mechanism, and the tabular NAS benchmarks as the place where the claim can be checked exactly.
Research Program
Representation. Every candidate, whether a cell in a tabular benchmark or a free-form PyTorch module written by the language model, is traced to an operator graph. The graph is summarized as a descriptor vector, chosen deliberately to resemble chemical descriptors. It contains:
- counts of each operator type,
- depth and width statistics,
- parameter count and activation memory,
- skip-connection density,
- Weisfeiler–Lehman subgraph hash counts, a standard way to fingerprint small graph motifs, analogous to Morgan fingerprints for molecules.
Using hand-built descriptors is deliberate. Identifying domains of applicability of machine learning models for materials science warns that domain definitions depend on the feature space, so we will also compare against a learned graph-network embedding.
Surrogate. We fit an XGBoost or graph-network predictor $f$ on all candidates trained so far, following the NAS-Bench-301 recipe.
Domain score. Each candidate $c$ gets a score $a(c)$ that combines three BAARD-style terms:
- Global: the Mahalanobis distance of $c$'s descriptors from the trained set, as used in solubility AD work.
- Local: the mean distance to $c$'s $k$ nearest trained neighbours.
- Decidability: the gap between $f(c)$ and the true accuracies of those $k$ neighbours.
The three terms are combined by a logistic meta-classifier, as in BAARD. It is trained to predict whether the surrogate's absolute error exceeds a tolerance $\varepsilon$.
Gating rule. The rule is $$\text{fitness}(c) = \begin{cases} f(c) & \text{if } a(c) \le \tau \\ \text{true trained accuracy of } c & \text{otherwise,} \end{cases}$$ where $\tau$ is recalibrated after every batch. Calibration uses held-out trained candidates: $\tau$ is the largest threshold at which the 90th-percentile surrogate error inside the domain stays below $\varepsilon$. Trained candidates are added to the dataset, so the domain grows exactly where the search goes. This addresses the static-boundary limitation of classical applicability domains. As in standard practice, the final top-$k$ architectures are always trained for real.
The language model never sees the gate. Its proposals are unconstrained, and only their scoring changes.
Measuring exploitation. The central measurement is selected-candidate error: surrogate prediction minus true accuracy, averaged over the candidates the optimizer keeps, compared with the same quantity over random candidates. If selected-candidate error is not larger than random-candidate error, the exploitation premise fails and the project reduces to a compute-saving heuristic.
Baselines
Train everything. The naive reference is to train every candidate, as AgentNAS does by default. It defines the accuracy ceiling for a given search trajectory and the cost the gate must undercut.
AgentNAS itself. On the open-ended tasks, the strongest existing system is AgentNAS. It reports an average error of 13.06% on the seven Unseen NAS tasks, against 15.04% for the best conventional NAS baseline, and a Spherical error of 38.71%. We run it with the same language model, the same GPU-hour budget and the same slotted search. Our variant replaces some real training runs with gated predictions and spends the savings on more candidates. Beating it means lower test error at equal GPU hours, or equal error with a large fraction of the training budget saved, on the Unseen NAS and NAS-Bench-360 tasks.
Ungated surrogate. Always trusting the surrogate isolates the value of the gate. On NAS-Bench-301 its held-out quality is a rank correlation of about 0.8. We expect it to degrade under optimization.
Uncertainty-aware alternatives. Three competitors are run at the same evaluation budget with the same surrogate family and the same tuning budget for thresholds:
- Ensemble-disagreement gating: train five surrogates and train a candidate for real when their spread is large.
- OMNI: substitute zero-cost proxies and learning-curve extrapolation for the gate.
- Penalty variant: the Rittig-style approach, which uses the domain score to penalize out-of-domain candidates rather than route them to training.
Ablations. Two ablations isolate the key ingredients. The first drops the decidability term and keeps distance only. The second replaces handcrafted descriptors with a learned embedding.
Why not just…?
Why not just use ensemble disagreement as the uncertainty signal? This is the most serious threat. Ensembles are the default way to make a surrogate uncertainty-aware. The open question is whether ensemble members trained on the same biased sample disagree in the regions the optimizer exploits, or stay wrong together. We have no result in the file on this for architecture search. So we treat it as an empirical contest. The test is the detection AUROC for "error > $\varepsilon$" computed specifically on optimizer-selected candidates, alongside final regret. If ensembles match the distance gate, we report that and keep the gating protocol and the exploitation measurement as the contribution, using whichever signal wins, or both combined in the meta-classifier.
Why not just use zero-cost proxies or early stopping, which need no trust model? These signals are cheap per candidate. However, the predictor benchmark shows that standalone proxies rank architectures less well than hybrids (White et al.). Their reliability outside image classification is also unclear, given that NAS-Bench-360 finds near-zero ranking correlation between CIFAR-100 and tasks such as NinaPro and Darcy Flow. Proxies can also be gated, so we include OMNI both as a competitor and as an alternative $f$ inside our gate.
Why not just train a well-tuned Wide ResNet? NAS-Bench-360 shows that a tuned Wide ResNet often matches NAS at modest budgets. We include it as a fixed reference on every task. The proposal's value is conditional: it lowers the cost of search, and so moves the budget at which search starts to pay off. We report that crossover point directly.
Proposed Experiments
Controlled test with exact ground truth. On NAS-Bench-201 (15,625 architectures; best CIFAR-10 accuracy 94.37%) and on the NAS-Bench-301 surrogate spaces, every architecture's accuracy is known. That makes the exploitation and regret measurements exact. We run regularized evolution and a language-model mutation operator under every baseline. The surrogate is seeded deliberately from a biased initial sample, mimicking a language model's stylistic priors. We report:
- selected-candidate versus random-candidate error,
- the gate's detection AUROC on selected candidates,
- final true accuracy versus the number of real evaluations.
Latency check. We repeat the procedure with latency as the target on HW-NAS-Bench. There, FLOPs correlate with Edge TPU latency at a rank correlation as low as 0.18, so a latency surrogate faces a hard extrapolation problem. No reported result was found on Lacuna for applicability-domain-gated latency prediction.
Open-ended test. We then run the full code-writing loop on NAS-Bench-360 and the Unseen NAS tasks against AgentNAS at matched GPU hours. We pay special attention to DeepSEA, where we check whether candidates flagged out-of-domain are over-represented among those whose validation gains failed to reach test.
Decision criteria. The decisive outcome has two parts. Ungated surrogate search must show a growing selected-candidate error, which confirms the premise. And at equal real-training budget, the gated search must recover the train-everything accuracy while ensemble gating does not. If the first fails, the problem is not real in architecture search. If only the second fails, the domain machinery is unnecessary and ensembles suffice.
References
- Graph machine learning for design of high‐octane fuels
- Intriguing Usage of Applicability Domain: Lessons from Cheminformatics Applied to Adversarial Learning
- Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
- How Powerful are Performance Predictors in Neural Architecture Search?
- Agentic Neural Architecture Search
- Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
- POLO: An LLM-Powered Project-Level Code Performance Optimization Framework
- $A_2$DEPT: Large Language Model–Driven Automated Algorithm Design via Evolutionary Program Trees
- Identifying domains of applicability of machine learning models for materials science
- Towards the prediction of drug solubility in binary solvent mixtures at various temperatures using machine learning
- NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark