Reading Architecture Quality at the Moment Its Structure Settles

Methods that search automatically for a good neural network design, called neural architecture search, spend most of their compute on one question: how good would this candidate be if we trained it fully? Two cheap answers exist, and both fail in known ways. Scoring a candidate at random initialization, before any training, is nearly free. But the network's internal representation is poorly organized at that point, and the scores break down when the candidates differ in overall shape. Training each candidate briefly and reading its accuracy is more faithful but still expensive, and in practice it ranks candidates worse than the zero-training scores. Work on pruning points to a third moment. A network's on/off structure, meaning which units fire for which inputs and which weights matter, settles within the first few percent of training, long before its weights stop changing.

This proposal scores each candidate architecture at the moment its own structure settles. We train every candidate for a short, adaptive number of steps and stop it when its pattern of active ReLU units on a fixed probe batch stops changing. At that point we compute cheap quality scores on the partly trained network. The stopping time is also a signal in its own right. The claim to test is that this per-candidate "settling time" ranks architectures better than scores taken at initialization, at a cost of a few hundred gradient steps rather than epochs. A second question is whether the settling time grows predictably with dataset size, which would let the training budget be set in advance.

Core Claim

Let $a_t(x) \in \{0,1\}^N$ be the binary pattern of which of a network's $N$ ReLU units are active on input $x$ at training step $t$. We define the partition drift over a fixed probe batch $B$ as

$$d(t) = \frac{1}{|B|\,N} \sum_{x \in B} \big\| a_t(x) - a_{t-\Delta}(x) \big\|_1 ,$$

where $\Delta$ is a spacing of a few steps. A candidate's settling time $\tau$ is the first step at which $d(t)$ stays below a threshold $\epsilon$ for a window of $w$ checks. Our claim is that architecture-quality scores computed at $\tau$ correlate with final trained accuracy much better than the same scores computed at $t=0$. This includes activation-pattern diversity scores such as the NWOT score, and gradient-saliency scores such as Synflow and SNIP. The gain should be largest exactly where initialization-time scores are known to fail, which is across candidates with different overall shapes. A secondary claim is that $\tau$ grows roughly as $\log m$ in the number of training examples $m$, so the budget can be set from a formula instead of a meta-search.

Several papers motivate the design. Max-Affine Spline Insights Into Deep Network Pruning treats ReLU networks as piecewise-linear maps that divide input space into regions. It shows that the placement of region boundaries freezes early, and it tracks this with Hamming distance on activation patterns, which is the measurement we reuse. Drawing Early-Bird Tickets supplies the stopping mechanics: it tracks mask distance over a sliding window and prunes once the distance falls below a threshold. That paper compresses a single fixed network, however, and never asks whether the settling point ranks different architectures. How Much Pre-training Is Enough to Discover a Good Subnetwork? supplies the theoretical tension. It proves that for two-layer networks in the lazy, kernel-like training regime, the feature matrix is poorly conditioned at initialization. It also proves that the training needed before a good subnetwork appears scales as $O(\log m)$. If this carries over, even loosely, then initialization is the wrong time to score a network.

Other work constrains the design. Zero-Cost Proxies for Lightweight NAS scores untrained candidates with pruning-at-initialization saliency measures and reports that jacob_cov degrades on candidates with differing overall structure. Zero-Cost Proxies Meet Differentiable Architecture Search scores the operations of an untrained supernet by perturbation. Both work strictly at $t=0$, and we keep their scores but move the time at which they are read. Two pieces of prior work rule out the obvious alternatives. DARTS+ already stops differentiable search when the ranking of architecture weights holds steady over a 10-epoch window, so an "early-bird criterion on the architecture weights" would largely repeat it. We therefore work with standalone candidates and the internal structure of the network, not with supernet routing weights. SuperTickets jointly searches and prunes inside a supernet over full training schedules. It is the closest joint method, but it answers a different question, namely what compact network to deploy. We ask how to rank candidates cheaply.

Research Program

The first step is to build the settling-time estimator. For each candidate, we train with the benchmark's own recipe and a fixed seed. Every $\Delta$ steps we record activation patterns on a 256-image probe batch, alongside the channel magnitude mask that the Early-Bird tickets paper uses. We stop at $\tau$, with a hard cap of 2% of the full schedule. At $\tau$ we compute NWOT, Synflow, SNIP, grad_norm, and a vote ensemble of them. We also compute two structural features: $\tau$ itself, and the level at which $d(t)$ plateaus. We choose $\epsilon$ and $w$ once, on a held-out slice of NAS-Bench-101 CIFAR-10 candidates, and freeze them for every other benchmark. This directly addresses the threshold-tuning problem that Early-Bird work leaves open.

The second step is analysis. We measure Spearman rank correlation with final accuracy for each score read at $t=0$, at a fixed step $k$, and at $\tau$. We also examine how $\tau$ is distributed across operation types. For example, if cells rich in skip-connections and pooling settle trivially fast, $\tau$ could become a proxy for the number of parameters rather than for quality. To catch this, we report partial correlations that control for parameter count and FLOPs.

The third step tests the scaling claim. On CIFAR-10 subsets with $m \in \{5\text{k}, 10\text{k}, 25\text{k}, 50\text{k}\}$, we fit $\tau$ against $\log m$ and against a power law, per architecture family. The $\log m$ form comes from the two-layer kernel-regime bound. Deep cells lie outside that theory, so this is a genuine test and not an assumed result.

Baselines

The naive reference point is random search at equal wall-clock time, using the same NAS-Bench-201 and NAS-Bench-101 query protocol. NAS-Bench-201 found that random search often beats elaborate methods, and any score that fails to improve search over it is not worth the extra steps. We treat it as beaten if the regret curve, measured as the gap to the benchmark optimum over wall-clock time, is lower across the full budget.

The strongest existing competitor is the $t=0$ zero-cost vote ensemble of Synflow, jacob_cov, and SNIP from Zero-Cost Proxies for Lightweight NAS. That paper reports a Spearman rank correlation of 0.82 on NAS-Bench-201, against 0.61 for the reduced-training proxy EcoNAS. We will recompute the ensemble on exactly the same candidate subset, seeds, and probe batch as our method, so that the comparison is between reading times and not between implementations. Beating it means a higher correlation on all three NAS-Bench-201 datasets and on NAS-Bench-101. On NAS-Bench-101, the paper's "four times faster" search claim sets the bar for the downstream search.

For the reduced-training baseline, NAS-Bench-101 already contains 4-epoch and 12-epoch accuracies for every model. We can therefore compare against short-budget accuracy at no cost, and match its epoch count and wall-clock time to ours. We also include Rethinking Performance Estimation in Neural Architecture Search, which fits a random-forest search (called MIP) to find the minimum training budget that keeps rank correlation high. We run it with the same total compute, including the cost of its meta-search. The file gives only its CIFAR-10 error of 2.66% and its ImageNet transfer accuracy of 74.18%. It reports no rank correlation on NAS-Bench-201, so we will measure one.

The key ablation separates adaptive timing from simply training a little. It reads the same scores at a fixed step $k$, where $k$ is the median $\tau$ across the benchmark, so the average compute is identical. A second ablation replaces activation-pattern drift with magnitude-mask drift. This tests whether the region-partition view adds anything beyond ordinary Early-Bird masks.

Why not just…?

Why not just use the zero-cost scores at initialization, which cost one batch? For shared-structure spaces like NAS-Bench-201 they are already good, at 0.82, and the headroom there may be small. The case for waiting rests on two points. The first is the conditioning argument in How Much Pre-training Is Enough to Discover a Good Subnetwork?. The second is the reported failure of jacob_cov on diverse architectures. The decisive experiment is therefore on candidates with differing overall structure. If our method only matches $t=0$ scores on NAS-Bench-201 and does not win on the heterogeneous set, the extra steps are not justified, and we will say so.

Why not just train every candidate for a fixed few epochs and read accuracy? Evidence already weighs against this. EcoNAS reaches only 0.61 correlation, and the zero-cost paper documents that a 64-fold FLOP reduction yields only about a 4-fold wall-clock speedup. Short-budget accuracy is dominated by how fast a candidate optimizes, not by its final quality. The NAS-Bench-101 4- and 12-epoch tables test this directly at matched cost.

Why not just recompute zero-cost scores after a fixed small number of steps, with no adaptive stopping? This is the real threat, and it is exactly the fixed-$k$ ablation. If fixed $k$ matches reading at $\tau$, the contribution shrinks to a simpler finding: scores read after a few hundred steps beat scores at initialization, and the $\log m$ fit tells you what $k$ to use. That finding would still be useful, and we would report it as the result rather than defend adaptive stopping.

Why not just apply DARTS+-style stability stopping to a supernet? DARTS+ already does this for supernet architecture weights. Supernets also suffer from poor global rank correlation, as shown in Rethinking Performance Estimation. Standalone short runs avoid shared-weight interference, which is why we measure the network's internal structure rather than the routing weights.

Proposed Experiments

The correlation study covers 3,000 stratified NAS-Bench-201 candidates on each of CIFAR-10, CIFAR-100, and ImageNet-16-120, plus 3,000 NAS-Bench-101 candidates. Each NAS-Bench-201 cell is small, so a few hundred steps per candidate keeps the study to a few GPU-days. We report Spearman and Kendall correlation, top-10% retrieval, and wall-clock time per candidate. Wall-clock time matters because of the documented gap between FLOP savings and GPU time.

For downstream search, we plug each score into random search and regularized evolution, both as a warm-up filter and as a way to choose which mutations to evaluate, following the protocol of the zero-cost proxies paper. We report final test accuracy against the benchmark optima of 94.37%, 73.51%, and 47.31% on NAS-Bench-201, and 94.32% on NAS-Bench-101, as a function of total compute.

The heterogeneity test uses a pool of PyTorchCV-style models with differing overall structure, trained for real. No reported result exists on Lacuna for zero-cost correlation on PyTorchCV, so both methods are measured fresh on the same pool.

Three checks decide whether the claim holds. The first is whether reading at $\tau$ beats reading at $t=0$ on the heterogeneous pool. The second is whether it beats the fixed-$k$ ablation at equal compute. The third is whether $\tau$ follows $\log m$ closely enough to predict the budget on a held-out dataset size to within one probe interval.