Many image tasks are rotationally symmetric at small scales and asymmetric at large ones. A galaxy's local texture looks the same at any angle, but its spiral arms have a handedness. A "6" is built from the same strokes as a "9", yet the two digits are distinguished only by orientation. Networks that hard-code rotation symmetry in every layer handle this badly. On the Galaxy10 DECals dataset, a network forced to respect the eight rotations and flips of a square (the group $D_4$) in all layers reaches 50.8% test error. Letting an evolutionary search relax symmetry layer by layer brings that down to 20.3% (Equivariance-aware Architectural Optimization of Neural Networks). That search is expensive, though. It trains every mutated candidate for a short horizon, and its differentiable variant keeps one parallel convolution per symmetry group in GPU memory.
We propose to decide, before any training, how much symmetry each layer should keep. We start from a network that is fully symmetric at initialization and take a few backward passes on real data. For each layer, we measure how much of the loss gradient points in directions that would break the symmetry. A basic fact about equivariant networks makes this well defined. If the network is exactly $D_4$-invariant and the labelled data distribution is also invariant, the expected gradient lies entirely inside the symmetric parameter subspace. Any leftover symmetry-breaking gradient is therefore a direct, per-layer readout of how much the task wants that layer's symmetry broken. One set of gradients scores every layer-wise symmetry assignment at once. The cost is a handful of minibatches, independent of how many candidates the search space contains.
Core Claim
The claim has three parts. First, the per-layer "symmetry-breaking gradient energy" of a fully $D_4$-equivariant network at initialization predicts which layer-wise subgroup assignment will generalize best after full training. Second, it does this better than the existing training-free architecture scores, known as zero-cost proxies. Those scores were designed for unconstrained cells and are biased by raw parameter count (NAS-Bench-Suite-Zero). They should systematically misread the weight tying that equivariance imposes. Third, the new score matches the architectures found by short-horizon evolutionary search at a small fraction of the cost.
The claim is only interesting if it also beats a much cheaper idea. That idea is to adopt the "local equivariance trend" reported by Towards Architectural Optimization of Equivariant Neural Networks over Subgroups, in which early layers want $D_4$ and late layers want no rotation symmetry, and then tune a single cut depth. We treat that heuristic as the baseline to beat, not a footnote.
The Mechanism
Take a network built with the equivariance-relaxation reparameterization of Equivariance-aware Architectural Optimization of Neural Networks. In that construction, each layer's kernel $\theta_k$ lives in the unconstrained (translation-only) parameter space, and a $D_4$ layer is simply a point where $\theta_k$ is fixed by the group action. Let $P_G$ be the orthogonal projection onto kernels fixed by subgroup $G$, with $G$ ranging over the chain $D_4 \supset C_4 \supset C_2 \supset C_1$. Here $C_n$ is the group of $n$ rotations and $C_1$ is no rotation symmetry. Let $\mu_k = \mathbb{E}[\nabla_{\theta_k} L]$ be the expected loss gradient at layer $k$, evaluated at a random $D_4$-symmetric initialization.
Rotating the input of a group convolution is equivalent to rotating its filters. At a symmetric parameter point, the per-example gradient therefore transforms by the group action on parameters. Averaging over a label-invariant data distribution then forces $\mu_k = P_{D_4}\mu_k$. For $D_4$ this holds exactly on a pixel grid, because 90° rotations and flips permute pixels without interpolation. This matters because it sidesteps the aliasing that The Lie Derivative for Measuring Learned Equivariance identifies as the dominant source of broken equivariance under continuous rotations.
The score for relaxing layer $k$ from $G$ down to $G'$ is the gradient energy that becomes available only after relaxation. It is the squared norm of $(P_{G'}-P_G)\mu_k$. Minibatch noise alone would make this positive even for perfectly symmetric data. We therefore estimate it without bias from $M$ independent minibatch gradients $g_1,\dots,g_M$:
$$\hat B_k(G \to G') = \frac{1}{M(M-1)}\sum_{m\neq m'} \big\langle (P_{G'}-P_G)\,g_m,\ (P_{G'}-P_G)\,g_{m'} \big\rangle .$$
The cross terms cancel the noise contribution, so on symmetric data the estimate is near zero regardless of batch size.
A candidate architecture $a$ assigns a subgroup $G_k(a)$ to each layer $k$. We score it as the total first-order loss decrease it can access, minus a complexity charge on its free (untied) parameters:
$$S(a) = \sum_k \hat B_k\big(D_4 \to G_k(a)\big) - \lambda\,\big[\,n_{\text{free}}(a) - n_{\text{free}}(D_4)\,\big].$$
Here $n_{\text{free}}$ counts independent parameters after tying. The single trade-off weight $\lambda$ is set once on one dataset and then frozen. The score is additive across layers because it is evaluated at the all-$D_4$ point. That additivity is an approximation, since relaxing one layer changes the gradients of the others. We test it directly with a sequential variant that recomputes the scores after each relaxation.
Research Program
The first deliverable is the missing measuring stick: a small tabular benchmark of equivariant architectures. Without full training results for every candidate, nobody can measure proxy rank correlation in this setting. The file records no reported zero-cost proxy result on any equivariant search space.
The backbone is a six-layer group-convolutional network following the EquiNAS setup. Each layer takes one subgroup from $\{D_4, C_4, C_2, C_1\}$, at two channel widths. Enumerating all $4^6 = 4096$ assignments is too costly. We therefore fully train all assignments whose symmetry is non-increasing with depth, which is 84 per width. We add 100 uniformly sampled non-monotone assignments, so the benchmark does not bake in the trend it is meant to test. Each configuration is trained to convergence with three seeds on four datasets. Galaxy10 DECals and ISIC 2019 carry imperfect real-world symmetry. RotMNIST is globally symmetric except for the 6/9 ambiguity. The fourth is a controlled MNIST family in which a fraction $p$ of training images is randomly rotated, so global asymmetry can be dialled from $p=0$ (fully upright) to $p=1$.
The second deliverable is the proxy and its analysis. We will check three things. First, that $\hat B_k$ is near zero on symmetric data and grows with the asymmetry dial $p$. Second, that its depth profile reproduces the layer-wise preferences EquiNAS found by search. Third, how many minibatches $M$ are needed before rankings stabilize.
We will also test the one real risk to the mechanism. At random initialization, features are uninformative, and the symmetry-breaking signal may only emerge once the network has learned something. To probe this, we compute $S$ at initialization and after one epoch of all-$D_4$ training. We will report how much ranking quality this minimal warm-up buys, and at what cost.
Baselines
The naive references are the two uniform networks, all-$C_1$ and all-$D_4$, trained on the same data and schedule as every candidate. Towards Architectural Optimization of Equivariant Neural Networks over Subgroups reports 31.6% test error for $C_1$ and 50.8% for $D_4$ on Galaxy10. Our reimplementation must land near these numbers before any comparison is trusted. We also include a plain $C_1$ network trained with random $D_4$ augmentation, because that is what a practitioner would try first.
The strongest existing method is EquiNAS$_E$ from Equivariance-aware Architectural Optimization of Neural Networks. It is greedy hill-climbing over relaxation morphisms, with short-horizon training of each mutation, and it reports 20.3% test error on Galaxy10. Its differentiable sibling EquiNAS$_D$ reports 21.8%, and the soft-constraint method Residual Pathway Priors reports 24.3%, both in Towards Architectural Optimization of Equivariant Neural Networks over Subgroups. We rerun EquiNAS$_E$ on our backbone and datasets with its published horizon. We log total GPU-hours and select the final architecture on the same validation split. Every selected architecture, ours included, is then retrained from scratch under one identical schedule with three seeds.
Beating EquiNAS$_E$ means one of two outcomes. Either our architecture's final error is within one seed-standard-deviation of theirs at no more than one tenth of their search compute, or it is strictly better at equal compute once the proxy's savings are spent on a slightly wider backbone. The ISIC 2019 comparison has no reported number in the file, so there we compare only against our own reruns.
The standard zero-cost proxies are synflow, grad_norm, snip, jacob_cov/nwot, and their vote ensemble, implemented as in Zero-Cost Proxies for Lightweight NAS. We evaluate them on the tabular benchmark in two forms: as published, and recomputed over free parameters so that tying is not double-counted. The published reference points come from unconstrained cells. The vote ensemble reaches Spearman 0.82 on NAS-Bench-201 (Zero-Cost Proxies for Lightweight NAS), and synflow and NASWOT reach Kendall tau 0.57 on NAS-Bench-201 CIFAR-10 (EZNAS). No such number exists for equivariant search spaces. Our proxy must beat the tying-corrected versions in Kendall tau over the full benchmark and in top-10% recall, since tail discrimination is where EZNAS shows these proxies fail.
Two ablations isolate the key ingredient. The first replaces $\hat B_k$ with the naive squared norm of the averaged gradient, without noise debiasing. This tests whether the cross-batch estimator matters. The second keeps the structure of $S$ but scores the symmetry-breaking projection of a random unit vector instead of the gradient. This tests whether the data signal, rather than the parameter-count term, drives the ranking.
Why not just…?
Why not just tune the cut depth of the local equivariance trend? This is the serious threat. The trend was observed robustly across layers (Towards Architectural Optimization of Equivariant Neural Networks over Subgroups). If the optimum always has the form "$D_4$ up to layer $k$, $C_1$ after", then seven full trainings find it. Seven trainings is cheaper than EquiNAS$_E$ and needs no proxy. The tabular benchmark settles this directly, because it records whether the best architecture on each dataset lies on the one-parameter cut family or uses intermediate groups such as $C_4$ and $C_2$. If it always lies on the family, we will say so. The remaining contribution would then be narrower: predicting $k$ from gradients at initialization, and showing on the asymmetry-dial datasets that $k$ shifts with $p$ in the way $\hat B_k$ predicts.
Why not just use existing zero-cost proxies with a parameter-count correction? NAS-Bench-Suite-Zero shows that these proxies largely track parameter and FLOP counts. Once that is corrected, nothing in synflow or nwot refers to the data's symmetry. We therefore expect them to rank assignments identically on the $p=0$ and $p=1$ MNIST variants, even though the best architectures differ. That invariance to $p$ is a sharp prediction, and the benchmark tests it for the corrected versions as well.
Why not just train each candidate briefly, as EquiNAS$_E$ does? Short training is a real competitor, and it is included at matched compute. Two pieces of evidence suggest it has a structural bias. Constrained layers converge faster early in training (Towards Architectural Optimization of Equivariant Neural Networks over Subgroups). Short horizons can invert rankings between fast starters and high-ceiling models (Rethinking Performance Estimation in Neural Architecture Search). Our score is not immune to this, since it is also a first-order, early quantity. The benchmark will show which estimator's rank errors correlate more with the number of free parameters.
Why not just augment a plain network with rotations? On data that is globally symmetric and large, augmentation or scale may be enough. A fine-tuned MAE vision transformer reaches 1.14% on RotMNIST against 2.28% for a G-CNN (The Lie Derivative for Measuring Learned Equivariance). But augmentation imposes a single global symmetry and cannot express "symmetric locally, asymmetric globally". Full augmentation also destroys the 6/9 distinction outright. The augmented $C_1$ baseline on Galaxy10 and on intermediate values of $p$ measures how much of the gap it closes.
Proposed Experiments
The first check validates the mechanism on controlled data. On the MNIST asymmetry dial, $\hat B_k$ should be statistically indistinguishable from zero at $p=1$. It should rise as $p$ decreases, and the rise should concentrate in the last layers, where orientation becomes class-relevant. If $\hat B_k$ is large at $p=1$, the noise debiasing or the exactness argument has failed, and the idea stops there.
The second check measures ranking quality. On all four benchmark datasets, we report Kendall tau and top-10% recall against fully trained error for our score and for each proxy baseline. The central test is Galaxy10, with $\lambda$ fixed from MNIST. Success means Kendall tau above every tying-corrected zero-cost proxy and above EquiNAS$_E$'s short-horizon score computed over the same candidates.
The third check measures end-to-end search. We take the top-scoring architecture under a budget of $M$ minibatches, retrain it, and compare its test error and total search GPU-hours with the rerun EquiNAS$_E$ (published reference: 20.3% on Galaxy10), EquiNAS$_D$ (21.8%), and the tuned cut-depth heuristic.
The outcome that would make this worth publishing has three parts. The chosen architecture must match EquiNAS$_E$ within seed noise on Galaxy10 and ISIC 2019, at least an order of magnitude cheaper. It must use a non-trivial intermediate subgroup somewhere, or else predict the cut depth correctly with zero trainings. And it must beat that heuristic whenever the benchmark shows the optimum lies off the cut family.
References
- Equivariance-aware Architectural Optimization of Neural Networks
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
- Towards Architectural Optimization of Equivariant Neural Networks over Subgroups
- The Lie Derivative for Measuring Learned Equivariance
- Zero-Cost Proxies for Lightweight NAS
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Rethinking Performance Estimation in Neural Architecture Search