Architecture search spends most of its compute training candidate networks to find out how good they are. The usual shortcut is to train every candidate for the same small number of epochs and rank candidates by their early accuracy. That ranking often disagrees with the ranking after full training. We propose a different signal for deciding when to stop. Trained networks tend to split into groups of units that work together, called modules. Evidence from language transformers suggests these groups settle into place long before training ends. The rule is simple: keep training a candidate until its modular structure stops changing, then score it. Candidates whose structure is still reorganizing get more epochs. Candidates that have settled are stopped early.
The bet is that the architectures misranked by fixed-budget early stopping are mostly the late bloomers, meaning networks whose internal organization is still unsettled at the cutoff. If that is true, a cheap structural check can spend the training budget where the ranking is still uncertain. The check is computed from the weights alone, with no data and no extra forward passes. We would test this on NAS-Bench-201, a public benchmark where all 15,625 architectures in a small search space have been fully trained. Because the true final ranking is known there, the claim can be confirmed or refuted directly.
Core Claim
Rethinking Performance Estimation in Neural Architecture Search shows that the number of training epochs matters most for how faithfully a cheap evaluation ranks architectures. It also shows that short-budget scores can rank well against each other at an early epoch yet correlate weakly with accuracy after full training. On the specialization side, Emergent Modularity in Pre-trained Transformers tracks clusters of feed-forward neurons during T5 pre-training. It finds that the coarse cluster structure stabilizes at about 15% of training, while the tuning of individual neurons continues until about 75%. Detecting Modularity in Deep Neural Networks shows that such clusters can be found by spectral partitioning of the weight matrices alone. Spectral partitioning here means splitting the graph of units and weighted connections into tightly connected groups using its eigenvectors. The clusters it finds this way are functionally important when lesioned, with a Fisher statistic of 2.54 for VGG-16 on ImageNet.
We claim the following. The epoch at which a candidate's weight-graph partition stops changing is a per-architecture signal that the candidate's accuracy rank has become trustworthy. An adaptive per-candidate budget built on this signal should reach the rank fidelity of a much longer fixed budget at lower average cost.
None of the cited work does this. The modularity papers use cluster structure only as a post-hoc diagnostic on finished networks. The budgeted-estimation work uses one global budget for all candidates. The novelty is to use structural settling as a stopping criterion that differs from candidate to candidate.
Research Program
The measurement works as follows. At checkpoint $t$, build a graph for a candidate network. Nodes are channels. The edge weight between an input channel and an output channel is the norm of the corresponding convolution kernel slice. Parameter-free operations are handled explicitly: skip connections add identity edges between matched channels, average pooling adds uniform edges, and the zero operation adds nothing.
Handling these operations is a real design problem rather than a formality. Prior modularity analyses concentrate on MLP and feed-forward blocks, as the file notes for Emergent Modularity in Pre-trained Transformers. NAS-Bench-201 cells are small branching graphs full of skip and pooling edges. We therefore also test a variant that drops parameter-free edges, so that the conclusions do not depend on this one choice.
We then partition the graph with normalized spectral clustering as in Detecting Modularity in Deep Neural Networks. We compare the partition at checkpoint $t$ with the partition from an earlier checkpoint using the adjusted Rand index, a standard score for how similar two clusterings are. The stopping rule is
$$t^\star = \min\{t : \mathrm{ARI}(P_t, P_{t-\Delta}) \ge \theta \text{ at two consecutive checks}\}, \quad t^\star \le T_{\max},$$
where $P_t$ is the partition at epoch $t$, $\Delta$ is the gap between checks in epochs, $\theta$ is a similarity threshold, and $T_{\max}$ is a hard cap on epochs. The candidate is then scored by its validation accuracy at $t^\star$. The partition costs one eigendecomposition of a graph with a few thousand nodes. That is small next to one training epoch and involves no data.
This choice is deliberate. Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient tracks specialization through a geometric complexity measure estimated with Langevin sampling, a noisy gradient-based sampling procedure. The authors report that it takes repeated passes over data and is tractable only for small models. That rules it out as something to run on thousands of candidates, so it constrains us rather than supplying our method.
We do not modify weight-sharing supernets. Analyzing and Mitigating Interference in Neural Architecture Search addresses a different cost problem, namely conflicting gradients on weights shared across candidates. Our proposal applies to stand-alone candidate training, and it can be plugged into any search algorithm that trains candidates.
The program has three stages. The first stage is descriptive. We ask whether weight-graph partitions in small image convolutional networks settle at all, and when. The 15% figure comes from language transformers and may not transfer. The second stage is predictive. For each architecture we compute its settling epoch and its rank error under fixed early stopping, and test whether the two are associated. The third stage is operational. We run the adaptive rule inside regularized evolution, a simple evolutionary search that repeatedly mutates good candidates and discards the oldest, and we measure what architecture it finds and at what cost.
Baselines
The naive reference point is full training. NAS-Bench-201 reports the true final test accuracy after 200 epochs for every architecture, and this defines the ground-truth ranking. The best architectures reach 94.37% on CIFAR-10, 73.51% on CIFAR-100 and 47.31% on ImageNet-16-120 (NAS-Bench-201). A random ranking gives the floor.
The strongest competitor on this exact question is fixed-budget early stopping, in the spirit of budgeted performance estimation from Rethinking Performance Estimation in Neural Architecture Search. That work trains every candidate under a reduced, uniform recipe. Combined with evolution, it reached 2.66% CIFAR-10 error in 0.33 GPU days, compared with 3,150 GPU days for AmoebaNet. Those numbers come from a DARTS-style space, not NAS-Bench-201. We found no reported result on Lacuna for the Kendall rank correlation of fixed-epoch early stopping against final accuracy on NAS-Bench-201, so we will measure it ourselves.
For fairness, both methods use the same retrained runs, seeds, learning-rate schedule and checkpoints. Fixed budgets are swept over a range of epoch counts, which traces a curve of rank correlation against average GPU hours. The adaptive rule is swept over $\theta$ and $\Delta$, with the same number of settings for each method, and tuned on a held-out set of architectures. The adaptive rule beats early stopping only if its curve lies above the fixed-budget curve. Concretely, it must reach higher Kendall correlation and higher top-10% precision at matched average epochs per candidate.
The strongest cost-free competitor is the family of zero-cost proxies, which score an untrained network from a single forward and backward pass. EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring reports a Kendall correlation of 0.65 on NAS-Bench-201 CIFAR-10 for its evolved proxy EZNAS-A, against 0.57 for NASWOT and 0.57 for Synflow. These proxies anchor the zero-epoch end of the cost curve. The same paper notes that no zero-cost proxy reliably separates the top 10% of architectures, which is where we expect an adaptive budget to help most.
The key ablation replaces partition stability with the simplest structural signal available: stop when the relative change in weight norm between checks falls below a threshold. A second ablation stops when a candidate's own validation accuracy plateaus. Both use the same sweep budget. If either matches the partition rule, the modular structure itself adds nothing, and we would report that.
Why not just…?
Why not just train everything for a fixed short budget? This is the main threat, and we do not dismiss it. The case against it is the gap its own authors document: rankings at an early epoch correlate well with each other but poorly with final accuracy (Rethinking Performance Estimation in Neural Architecture Search). A uniform budget cannot tell which candidates are the unreliable ones. The stage-two analysis settles this directly. If the architectures misranked at a fixed budget are not disproportionately those with late settling epochs, the proposal fails and fixed budgets win.
Why not just use a zero-cost proxy? Proxies are far cheaper. But the best one reported here reaches a Kendall correlation of 0.65, and proxies are known to fail among the top 10% (EZNAS). A search only needs a good ranking among the top 10%, so that failure is the costly one. Our experiments also test a hybrid that uses the proxy to discard the bottom half and the adaptive budget on the rest. That checks whether the two are complementary rather than competing.
Why not just stop when validation accuracy or weight change plateaus? These are simpler per-candidate rules and a real alternative, so they are included as ablations under equal tuning. The reason to expect partitions to differ is the result in Emergent Modularity in Pre-trained Transformers: coarse structure settles while individual units keep changing. That would mean partitions settle earlier than weight norms or accuracy curves. If this ordering does not appear in convolutional networks, the simpler rules should be preferred.
Why not just use random search or regularized evolution? They are search strategies, not evaluation methods. They look cheap on NAS-Bench-201 only because accuracy is read from a lookup table (NAS-Bench-201). Our rule is meant to be used inside them, and we evaluate it that way.
Proposed Experiments
We retrain about 1,000 NAS-Bench-201 architectures on each of CIFAR-10, CIFAR-100 and ImageNet-16-120. The architectures are stratified across final-accuracy deciles and oversampled in the top decile, and each is trained with the benchmark's 200-epoch recipe and a checkpoint every epoch. This retraining is needed because the benchmark stores accuracy logs but not intermediate weights. Our retrained final accuracies must reproduce the table's values before anything else is analyzed.
The first decisive check is whether partitions settle well before accuracy plateaus. The quantity of interest is the distribution of settling epochs as a fraction of 200 epochs, compared with the weight-norm and accuracy-plateau epochs. If partitions do not settle earlier, we stop and report a negative result.
The second check is the rank correlation between settling epoch and each architecture's rank error under fixed budgets of 5, 10, 25 and 50 epochs.
The third check is the cost curve described under Baselines. It is evaluated on held-out architectures and reports Kendall correlation, top-10% precision and average GPU hours per candidate.
The last check is end to end. Regularized evolution runs with each evaluation rule under matched total GPU hours, and we report the true final test accuracy of the returned architecture against the benchmark optimum, averaged over seeds. The result also needs to transfer across datasets: a threshold tuned on CIFAR-10 should be applied unchanged to CIFAR-100 and ImageNet-16-120.
References
- Rethinking Performance Estimation in Neural Architecture Search
- Emergent Modularity in Pre-trained Transformers
- Detecting Modularity in Deep Neural Networks
- Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
- Analyzing and Mitigating Interference in Neural Architecture Search
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring