Conservation-Drift Exponents for Cheap Architecture Ranking

Many neural networks contain quantities that plain gradient descent, taken with infinitely small steps, would never change. One example is the length of a weight vector that feeds into a batch-normalization layer. Real training uses finite steps, and those quantities then drift. The drift has an exact algebraic form, and over a fixed stretch of "training time" it should shrink in exact proportion to the learning rate, with an exponent of precisely 1, as long as the network trains the way small-step theory says it should. We propose to score candidate architectures by running a few dozen steps at three learning rates and checking whether this exponent stays pinned at 1 at the learning rate the architecture will actually be trained with. The score is a near-binary stability test. It is not fitted, it is computed from ratios so it does not grow with model size, and it costs a few hundred gradient steps rather than full training.

The idea is borrowed from condensed-matter physics. There, the hard problem of confirming a hidden symmetry at a phase transition was made tractable by measuring a quantity that must be an exact integer if the symmetry holds, rather than fitting noisy fractional exponents. Architecture search has a cost problem, and its cheap fixes are unreliable. Training every candidate is prohibitively expensive, and some estimates suggest the energy spent on the search can exceed what the resulting model saves in deployment (A Survey of Methods for Low-Power Deep Learning and Computer Vision). The cheap scores that replace training are heuristic, fragile across search spaces, and biased toward larger models (NAS-Bench-Suite-Zero). A score whose pass/fail value is fixed by theory rather than tuned would address both of those weaknesses at once.

Core Claim

Consider a group of weights $w$ whose output passes through a normalization layer, so the loss does not change when $w$ is rescaled. Rescaling invariance forces the gradient $g$ to be orthogonal to $w$. A single SGD step with learning rate $\eta$ therefore changes the squared norm by exactly

$$\|w_{t+1}\|^2 - \|w_t\|^2 = \eta^2 \|g_t\|^2 .$$

The same identity holds for the balance quantity $\|a\|^2 - \|b\|^2$ between the incoming weights $a$ and outgoing weights $b$ of a ReLU unit, with $\|g_a\|^2 - \|g_b\|^2$ in place of $\|g_t\|^2$. Under true gradient flow these quantities are exactly conserved (Abide by the law and follow the flow).

Now fix a training time $\tau = \eta T$, where $T$ is the number of steps. Define the accumulated relative drift $D(\eta) = \sum_{t<T} \eta^2 \|g_t\|^2 / \|w_0\|^2$, aggregated over all symmetric weight groups. We then have $D(\eta) \approx \eta \int_0^\tau \|g\|^2 d\tau'/\|w_0\|^2$. The exponent

$$p(\eta) = \frac{d \log D}{d \log \eta}$$

therefore equals 1 whenever the discrete trajectory tracks the continuous flow, because the integral then converges to its flow value. When the step size is too large for the architecture's loss surface, the iterates oscillate, gradient norms along the trajectory grow, and $p$ rises above 1.

Our claim has three parts. First, the deviation $|p(\eta_0) - 1|$, measured at the benchmark's own training learning rate $\eta_0$, predicts final accuracy. Second, it carries information not contained in parameter count or in short-horizon training loss. Third, because every quantity is a ratio, it is free of the size bias that afflicts gradient-magnitude scores.

Relation to Prior Work

Two physics papers supply the methodological lens, not the method. Both identify emergent symmetries at a phase transition by measuring the scaling of a conserved current, whose value is pinned by Noether's theorem, instead of fitting critical exponents. Emergent symmetry and conserved current at a one-dimensional incarnation of deconfined quantum critical point reports a current scaling dimension of 1.00(5) against the exact prediction of 1. Role of Noether's Theorem at the Deconfined Quantum Critical Point reports anomalous dimensions of 0.002(9) and 0.004(6) against an exact 0. We take from them the design principle: test for a pinned value rather than regress a free one.

We also take one of their cautions. The latter paper stresses "pseudocriticality," a weakly discontinuous transition that looks continuous up to the largest system size simulated. The analogue here is an architecture that tracks the flow over a short horizon and destabilizes later. We test this directly with a horizon ablation.

Abide by the law and follow the flow supplies the method. It characterizes all conservation laws of gradient flow through the Jacobian of the network's parameterization, and it provides a SageMath solver for small architectures. That paper studies conservation only in continuous time and never uses it to compare architectures. We use its solver to verify that the rescaling and normalization invariances we track are the complete set for the NAS-Bench-201 operations, so that no conserved quantity is silently missed.

Two lines of work use symmetry in a different way, and the proposal differs from both. Noether Networks learns conserved quantities of the data and enforces them through gradient steps at inference time, which adds cost rather than removing it. Equivariance-aware Architectural Optimization searches over discrete rotation and reflection groups acting on images, using evolutionary or supernet training. Neither examines the symmetries of the parameters under training, and neither reduces evaluation cost.

On the cost side, three works set the reference points. Zero-Cost Proxies for Lightweight NAS and EZNAS score networks at initialization using hand-designed or evolved formulas. NAS-Bench-Suite-Zero shows these scores are confounded by size and are strongest when combined in a learned model.

One paper constrains our claims. DARTS- shows that large Hessian eigenvalues occur even in strong architectures and so fail as a collapse indicator. Because $p > 1$ is at bottom a cheap signal of step-size instability, which is closely related to curvature, we make no claim about diagnosing supernet collapse. Our claim is limited to ranking stand-alone architectures, and we include a Hessian baseline to show whether the drift test simply reproduces it.

Research Program

We will build a small library on top of NASLib. For any candidate cell it will do three things. It will detect symmetric weight groups structurally, meaning weights feeding normalization layers and positively homogeneous unit pairs, and it will cross-check completeness against the SageMath solver on the NAS-Bench-201 operation set. It will run three short training runs at $\eta_0$, $\eta_0/2$ and $\eta_0/4$ to a common time $\tau$, using identical minibatch sequences so that gradient noise is shared across runs. It will then report $p(\eta_0)$ from the log-log slope of $D$.

With $K$ steps at $\eta_0$, the total cost is $7K$ steps. We will sweep $K$ from 10 to 100 to trace cost against rank quality. Architectures with no trainable path, such as all-zeroize cells in NAS-Bench-201, receive the lowest score by rule.

We will also compute a zero-cost variant: the one-step relative drift at initialization, $\eta_0^2 \|g\|^2/\|w\|^2$ per group. The purpose is to isolate what, if anything, the multi-rate exponent adds beyond a single gradient.

Baselines

The naive reference points are random ranking and parameter count. NAS-Bench-Suite-Zero identifies FLOPs and nwot (a score based on how differently a random network's activations respond to different inputs) as the most robust individual scores across tasks, though we found no single reported correlation value for them on Lacuna. A new score that cannot beat parameter count within size-matched bins has learned nothing beyond size.

The strongest existing zero-cost method is the vote ensemble of Synflow, Jacobian covariance, and SNIP from Zero-Cost Proxies for Lightweight NAS. That paper reports a Spearman rank correlation of 0.82 with final accuracy on NAS-Bench-201. We will recompute the ensemble in NASLib on the same architectures and seeds rather than rely on the quoted figure. Beating it means exceeding its recomputed correlation on each of CIFAR-10, CIFAR-100 and ImageNet-16-120, and doing so by a margin larger than the seed-to-seed spread.

The strongest competitor at equal compute is short training. Zero-Cost Proxies for Lightweight NAS reports a correlation of 0.61 on NAS-Bench-201 for EcoNAS, a reduced-training estimate. We will run our own reduced-training baseline at exactly $7K$ steps, using the same data order and ranking by training loss or minibatch accuracy. This is the comparison the proposal must win, because it spends the same budget our test does.

The learned-combination baseline is the XGBoost surrogate over the 13 proxies in NAS-Bench-Suite-Zero, which that paper reports improves rank correlation by up to 42%. We will add $p$ as one extra feature. The test is whether held-out correlation improves when the training set of architectures is the same size.

The curvature baseline is the top Hessian eigenvalue at initialization, estimated by power iteration within the same wall-clock budget.

The key ablation is the one-step zero-cost drift, which is essentially a layer-wise ratio of gradient norm to weight norm. If it matches $p$, the multi-rate exponent adds nothing and the claim falls back to the simpler score.

Why not just…?

Why not just use the existing zero-cost ensemble or the XGBoost surrogate? These are strong. NAS-Bench-Suite-Zero documents that their component scores rise with parameter count and need heuristic correction, and Zero-Cost Proxies for Lightweight NAS shows individual scores failing across heterogeneous model families. Neither weakness is guaranteed to vanish with our score. The size-binned correlation and the TransNAS-Bench-101 and NinaPro transfers will show whether a ratio-based, theory-pinned score is actually more robust. If it is not, the surrogate-feature experiment still tells us whether $p$ adds complementary information.

Why not just train for the same few hundred steps and rank by loss? This is the most serious threat. The exponent test spends real training steps, and early loss may already capture the same stability information. Our argument is that early loss measures progress at a single step size, whereas $p$ measures whether that step size is stable for the architecture. Architectures that make fast initial progress while sitting near instability should separate the two scores. The decisive number is the partial Spearman correlation of $p$ with final accuracy after controlling for early loss and log parameter count. If it is indistinguishable from zero, we will report short training as sufficient and treat the exponent as a diagnostic rather than a ranking score.

Why not just use the ratio of gradient norm to weight norm at initialization? For weights that feed normalization layers, our zero-cost variant is exactly this ratio. It costs one backward pass and is included as the central ablation. The multi-rate exponent differs because it observes how drift responds to the step size along a trajectory, which a single gradient at initialization cannot reveal. If the ablation matches $p$, we will adopt the simpler score and say so.

Why not just track the Hessian's top eigenvalue? DARTS- shows that high curvature coexists with good architectures, and power iteration costs repeated second-order products. We expect $p$ to correlate with curvature scaled by the learning rate, not with raw curvature, and to be cheaper. Head-to-head correlations at equal wall-clock time will settle both points.

Proposed Experiments

The main experiment covers all 15,625 NAS-Bench-201 cells (NAS-Bench-201) on CIFAR-10, CIFAR-100 and ImageNet-16-120, with three seeds. We report Spearman and Kendall correlations, top-10% overlap, and correlation within parameter-count deciles, plus wall-clock cost per architecture. We also use each score to initialize a random-search pool and report how many table lookups are needed to reach the space's best test accuracy of 94.37% on CIFAR-10. A score that finds strong cells in fewer lookups than unguided random search is useful even if its global correlation is modest.

The transfer experiment runs across the NAS-Bench-Suite-Zero tasks: NAS-Bench-101, the NAS-Bench-301 subset, TransNAS-Bench-101, SVHN, NinaPro and Spherical-CIFAR-100. Here we test whether $p$ degrades less than the ensemble does outside CIFAR-like image data.

Three decisive checks decide the claim.

First, sanity. On architectures that train smoothly, the measured $p$ must sit within noise of 1 as $\eta$ shrinks. If it does not, the drift estimate is dominated by minibatch noise and the method fails at its foundation. Shared minibatch sequences across the three runs are meant to prevent this.

Second, marginal value. As described above, $p$ must retain nonzero partial correlation with final accuracy after controlling for early loss and size.

Third, the pseudocritical failure mode. We will compare $p$ measured at $K = 20$ with $p$ measured at $K = 100$ against architectures whose final training curves show late divergence. If short-horizon $p$ misses them systematically, the test has the same blind spot the physicists warned about, and we will report the horizon needed to close it.