Search Spaces That Rewrite Themselves: Execution-Guided Refinement of LLM-Built Architecture Spaces

Architecture search, the automatic design of neural network layouts, has always depended on a human-written "search space": a menu of layer choices from which the optimizer assembles candidates. Hand-built menus from vision do not carry over to audio, genomics, or physics. Recent work has a language model write the menu as code for each new task, and then a standard optimizer searches within it. In the best current system, that menu is written once and then frozen. Our proposal treats the search space as a program that gets debugged. As the optimizer runs, it produces evidence: which slot choices never matter, which combinations crash, and which training runs diverge or underfit. A language model reads that evidence and edits the search-space code. It prunes dead options, repairs combinations that crash, splits slots that look promising, and writes new operators where the search has stalled.

The second question is whether such a debugged space is worth reusing on other tasks. Past work shows that copying a winning architecture to a new task often does worse than random search. We test whether the refined space carries over better than the architecture does, and better than natural-language "design principles" do. The note makes one falsifiable claim and pairs it with a compute-matched test designed so that a larger frozen menu, or simply restarting the pipeline, could beat it.

Core Claim

The closest system is Agentic Neural Architecture Search (AgentNAS). It has an LLM agent write a seed model, rewrite it as a "slotted architecture" with 4–8 operator alternatives per slot, and then hand that space to regularized evolution. The handoff runs one way: once the slots exist, they never change. AgentNAS itself reports the symptoms a frozen space should produce. Weaker LLMs wrote slotted code that did not run. On DeepSea genomics, validation gains did not reach the test set.

Our claim is that search statistics are a debugging signal for the space itself. Those statistics are per-option effect estimates, crash tracebacks per slot combination, and learning-curve pathologies. Feeding them back to an LLM that is allowed only to edit the space, never to pick individual architectures, should produce two things. It should give lower final error at equal GPU budget than a frozen space. It should also give a space whose random samples are better, which is the sign that the space improved and not merely the search.

This design follows the finding in AgentNAS that LLMs are poor at fine-grained combinatorial optimization, so the choice of individual architectures stays with evolution. It also recovers part of the open-endedness argued for by EvoPrompting, because new operators can enter the space mid-search.

Research Program

The system runs the search in rounds. In each round, regularized evolution evaluates a fixed number of configurations using short-schedule training. After each round, a statistics module computes three signals:

  • per-option marginal importance, from a random-forest fit of validation score on the slot choices evaluated so far;
  • crash and non-finite-loss rates per option and per pair of options, with the offending tracebacks attached;
  • simple learning-curve labels (diverging, plateaued early, still improving) for the top configurations.

A language model receives these signals together with the current space code. It may apply only a small set of typed edits: prune an option, add an option by writing new operator code, split a slot into two, merge slots, or change a "glue" element such as a skip connection or normalization.

Every edited space goes through an execution gate before it is accepted. The gate samples configurations from the new space and runs a forward and backward pass on one batch, checking tensor shapes and finite gradients. Any failure triggers a traceback-driven repair loop. This gate borrows two mechanisms. From A₂DEPT, which evolves solver programs with a self-healing step that writes missing helper functions, we take dependency repair. A₂DEPT also documents the "partial migration" failure, where one edited component breaks the modules that depend on it. Our gate targets exactly that failure at the level of slot interfaces. From MEIC, which debugs Verilog with a rollback to the last stable state, we take rollback. If the round after an edit falls below the pre-edit best by more than seed noise, the edit is reverted.

Evaluations already paid for carry across edits whenever their encoding remains valid in the new space. This means refinement does not quietly buy extra compute by restarting the population.

For transfer, a refined space from a source task becomes the starting space on a target task, and refinement continues from there. This differs from Design Principle Transfer in NAS via LLMs (LAPT). LAPT distills architectures into linguistic principles and uses them to prune fixed cell spaces, so it cannot add operators or cross paradigms. It also differs from ε-MemEvo, which transfers natural-language tactics between program-evolution tasks through a bandit gate, has never touched neural architecture code, and showed that injecting prior material without a gate causes negative transfer. We borrow that negative-transfer lesson directly. Whether to transfer, and from which source, is chosen by the same Thompson-sampling gate, with "start fresh" as an explicit arm.

The motivating evidence for transfer at the space level comes from two benchmarks. TransNAS-Bench-101 found that direct architecture transfer can underperform random search. NAS-Bench-360 found near-zero ranking correlation between CIFAR-100 and tasks such as NinaPro or Darcy Flow. The constraining evidence is about cheap scoring. Evaluating Efficient Performance Estimators shows that zero-cost scores are topologically biased and often lose to parameter count. NAS-Bench-360 reports that these scores are volatile across domains. For that reason we use short-schedule training as the fidelity, and we use training-free scores only as a crash-and-degeneracy filter, not as a ranker.

Baselines

The naive reference point is a tuned Wide ResNet per task, as in the NAS-Bench-360 "buy versus build" comparison. It gets the same tuning budget in GPU-hours as the search methods. The file does not give a numeric value for it, so we will reproduce it; no reported result found on Lacuna for per-task tuned WRN error figures.

The strongest existing method is AgentNAS, run with its own seed and slotted-space pipeline, the same LLM, the same short-schedule fidelity, the same number of evaluations, and the same single-GPU memory cap. Its reported numbers are 38.71% error on NAS-Bench-360 Spherical (down from 45.08% for the LLM seed) and 13.06% average error on Unseen NAS versus 15.04% for the best conventional baseline (Agentic Neural Architecture Search). Beating it means lower mean test error across the 17 tasks, with a paired sign test over tasks and three search seeds each. Separately, it means not matching AgentNAS's validation–test divergence on DeepSea.

EvoPrompting-style direct code mutation serves as the open-ended competitor. We run it with the same traceback repair loop, so it is not handicapped by discarding broken code.

Two ablations isolate the key ingredient, which is the evidence fed to the editor. In the first, the editor makes blind edits with the space code only and no statistics. In the second, random typed edits are applied with the same execution gate. A third ablation removes rollback.

For transfer, the target-task comparisons are: fresh AgentNAS space, direct transfer of the source's best architecture, LAPT-style principle pruning applied to the fresh space, and the gated transfer of a refined space. All comparisons are compute-matched, including the source-task cost amortized across targets.

Why not just…?

Why not just have the LLM write a larger frozen space at the start? This is the most serious threat. If refinement mostly adds options, a space with twice as many options per slot might capture the same gain. We will run exactly that control, with slot count matched to the final size of our refined spaces, at equal evaluations. If it matches refinement, the claim fails. The random-sample quality metric below would also expose it: a wider frozen space should dilute sample quality, whereas pruning should concentrate it.

Why not just restart the AgentNAS pipeline several times and keep the best? Restarts buy diversity without feedback. We split the same budget into k independent AgentNAS runs with k chosen to match the number of our refinement rounds. Refinement must beat the best-of-k, not the average.

Why not just let the LLM mutate architectures directly, as EvoPrompting does? EvoPrompting trained every child from scratch and discarded broken code. AgentNAS argues that LLMs are weak fine-grained optimizers. With repair added so the comparison is fair, direct mutation isolates whether keeping a parameterized space matters at all. If direct mutation wins, the space abstraction is dead weight.

Why not just transfer principles as LAPT does? LAPT transfers well within cell spaces on TransNAS-Bench-101. However, its principles can only restrict a fixed menu. The transfer experiment runs LAPT pruning on the same starting space, so any advantage of refined spaces has to come from edits LAPT cannot express, such as added operators and repaired glue.

Proposed Experiments

The benchmarks are the 10 NAS-Bench-360 tasks and the 7 blind Unseen NAS tasks. Final architectures are retrained under a fixed recipe and reported on test error. The primary endpoint is mean test error at a fixed GPU-hour budget, with anytime error-versus-compute curves as a secondary endpoint.

The first decisive check separates "better space" from "better search." It samples 200 random configurations from the initial and final spaces of each run, trains each briefly, and compares the score distributions. Refinement should shift the upper quantiles upward. Blind and random edits should not.

The second check measures the execution gate: the fraction of edited spaces that run after repair, broken down by LLM strength. This tests whether gate plus rollback removes the brittleness AgentNAS reported with weaker models.

The third check is leave-one-task-out transfer within NAS-Bench-360. It reports target error and search speed for gated refined-space transfer against the fresh-start and direct-transfer baselines. It also reports how often the gate chooses "start fresh," which should happen for known outliers such as Darcy Flow, where NAS-Bench-360 found the expert Fourier Neural Operator strongest.

The outcome that would falsify the proposal is that the size-matched frozen space and the best-of-k restarts come within seed noise of refinement on most tasks. We would report that as the main result.