Equivariant networks build a symmetry, such as "rotating the input rotates the features," directly into their layers. When the data really has that symmetry, they learn faster and are more robust. Real data rarely has a symmetry everywhere, though. Recent architecture search work found that networks want strict symmetry in early layers and little or none in deep layers. Finding the right per-layer choice currently costs a full architecture search: either an evolutionary loop that trains many candidates, or a "super-network" that trains every symmetry option in parallel. This proposal tests a cheaper idea. Train one ordinary, unconstrained network for a few epochs. At each depth, measure how close its intermediate features already are to respecting each candidate transformation. Then build the final network by giving each layer the largest symmetry group its features already approximately obey. If this works, a measurement replaces a search.
The claim is specific and could easily be false. The per-depth "equivariance residual" profile of a briefly trained plain network should predict the layer-wise group assignment that full search finds, and the resulting architecture should match that search's test error on imperfectly symmetric data at a small fraction of its compute. Because the measurement only needs to apply a transformation to the input, it applies unchanged to continuous transformations such as arbitrary-angle rotation or scaling. Existing searches cannot represent those, because they choose only among a fixed list of discrete rotation and reflection groups.
Where this sits
The closest work is EquiNAS (Equivariance-aware Architectural Optimization of Neural Networks; Towards Architectural Optimization of Equivariant Neural Networks over Subgroups). It searches over discrete subgroups (cyclic $C_N$ and dihedral $D_N$) per layer, in two ways. The evolutionary variant uses a function-preserving "relaxation morphism," and the differentiable variant uses a mixed layer that sums convolutions constrained to each subgroup. EquiNAS motivates this proposal in two ways: it discovered the early-strict, late-relaxed "local equivariance trend," and it showed that layer-wise choices beat both rigid and unconstrained networks on Galaxy10. Our method differs in mechanism. EquiNAS trains candidates or a super-network and selects by validation accuracy. We train no candidates and select from a measured property of one plain network. Autoequivariant networks (Group Equivariant Neural Architecture Search via Group Decomposition and Reinforcement Learning) search group, augmentation and width with a reinforcement-learning agent that evaluates full training runs. That approach is even more expensive, and it too is limited to discrete groups. We borrow its capacity argument: weight sharing reduces the number of free parameters, so we match parameter counts when comparing.
The measurement itself is adapted from post-hoc symmetry extraction. LieGG (LieGG: Studying Learned Lie Group Generators) reads continuous symmetry generators off a trained network's input gradients, but only for fully trained models, only at the output, and only for connected matrix groups. LieGG also constrains our design. It reports that unconstrained networks show high "symmetry variance" in intermediate layers. Our probe may therefore be noisy, and the plan treats that noise as a central risk to measure. Zero-cost proxies (EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring) are the cost-reduction tradition we join. They score architectures at random initialization, but they have no notion of group action. They also struggle to separate top candidates, which is exactly the regime of choosing among good symmetry assignments. Random-weight results (On Random Weights and Unsupervised Feature Learning) show that the wiring of an untrained convolutional network alone produces translation invariance. That cuts against probing at initialization: an untrained network's invariances describe its architecture, not the data. For this reason our probe uses brief training, and we test the initialization variant only as an ablation.
Core Claim
For layer $l$ with features $\phi_l(x)$ and a candidate transformation $g$, define the equivariance residual
$$ r_l(g) = \min_{M} \frac{\mathbb{E}_x \lVert \phi_l(g\cdot x) - g\cdot M\,\phi_l(x) \rVert^2}{\mathbb{E}_x \lVert \phi_l(g\cdot x) \rVert^2}. $$
Here $g\cdot$ acts on the input image and on the spatial grid of the feature map, and $M$ is a learned channel-mixing matrix. A value of $r_l(g)$ near zero means the network up to depth $l$ already transforms predictably under $g$, so constraining layer $l$ to respect $g$ costs little. The minimization over $M$ is ordinary least squares on cached features, so computing the residual is cheap.
For a candidate group $G$, the layer score is the worst residual over its generators. For continuous groups, $g=\exp(\epsilon L)$ for a small $\epsilon$ and a generator $L$, in the style of Learning Lie Group Symmetry Transformations with Neural Networks. Each layer is assigned the largest group whose score falls below a threshold $\tau$, and $\tau$ is tuned once on validation data. The resulting network is then trained from scratch at a parameter count matched to the baselines.
The new contributions are this residual-profile selection rule and the claim that its cost is dominated by a few epochs of a single network.
Research Program
The first stage checks that the measurement means something, using a synthetic image task where the true symmetry varies with depth. Local patches are made rotationally symmetric, while the global arrangement of patches is oriented. We check that the residual profile jumps at the receptive-field size where symmetry breaks, and we record how many training epochs are needed before the profile stabilizes across seeds.
The second stage moves to the discrete groups EquiNAS can represent ($C_1, C_2, C_4, D_1, D_2, D_4$) on RotMNIST, Galaxy10 DECals and ISIC 2019. Here we compare assignments and error rates directly against the search.
The third stage adds continuous candidates, namely arbitrary-angle rotation and isotropic scaling via generators. We ask whether layers selected for continuous rotation equivariance beat the best discrete assignment on Galaxy10, where galaxy orientations are not aligned to a grid. This is the regime where search over a fixed group list has nothing to offer.
Throughout, we log total GPU hours including the probe, and measure how consistent the selected assignment is across probe seeds.
Baselines
The naive reference is a standard unconstrained convolutional network trained alone. On Galaxy10 it reaches 31.6% test error as reported in Equivariance-aware Architectural Optimization of Neural Networks. Alongside it we run the rigid all-layers $D_4$ network, which reaches 50.8% in the same paper and shows that over-constraining is a real failure mode. We include Residual Pathway Priors, reported there at 24.3%, as the strongest non-search approach to soft symmetry.
The strongest competitors are EquiNAS evolutionary search, at 20.3% test error on Galaxy10, and EquiNAS differentiable search, at 21.8%, both from the same paper. These are the methods our probe claims to replace, and they search exactly the space we select from. We rerun them with their published search spaces and with our data splits, augmentation and final-training recipe. We report their search compute alongside ours. No reported search-cost figure for EquiNAS was found on Lacuna, so we will measure it ourselves. Beating them means matching test error within seed noise, roughly within one point of 20.3%, at no more than a tenth of their measured search compute. A better error at equal compute would be a stronger result, but we do not expect it on discrete groups.
Two further baselines isolate the key ingredient. A compute-matched truncated EquiNAS differentiable search gets the same GPU hours as our probe. A hand rule from the local equivariance trend uses $D_4$ in the first half of the network and $C_1$ after, with the switch depth tuned on validation.
Ablations replace the briefly trained probe with the same probe at random initialization, and replace per-layer residuals with an output-only invariance score. NASWOT and Synflow, which reach a Kendall rank correlation of 0.57 on NASBench-201 CIFAR-10 in EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring, are used to rank candidate group assignments under the same budget.
For RotMNIST and ISIC 2019, no reported result was found on Lacuna for any of these methods. All numbers there will come from our own reruns. The $D_{16}$ result of 95.52% accuracy in E(2) Equivariant Neural Networks for Robust Galaxy Morphology Classification uses a different architecture and protocol, so we will cite it for context but not treat it as directly comparable.
Why not just…?
Why not just use the local equivariance trend as a fixed rule? This is the most serious threat. If "strict early, relaxed late" with one tuned switch depth matches EquiNAS, then neither search nor our probe is needed. We take this seriously and run that rule with its own validation-tuned switch. The probe earns its place only if it beats the rule on at least one dataset, or if the best switch depth or group differs across RotMNIST, Galaxy10 and ISIC in ways the probe predicts without validation sweeps. If the rule wins everywhere, we will report that as the finding.
Why not just run EquiNAS for fewer epochs? Budgeted estimation (Rethinking Performance Estimation in Neural Architecture Search) suggests short training often preserves rankings. EquiNAS also reports a specific bias: constrained branches get cleaner gradients and win early, so truncating its search should favour over-constraining. The compute-matched truncated search tests this directly. If it matches full search, our cost argument weakens to "both are cheap," though the continuous-group extension would still stand.
Why not just use existing zero-cost proxies? They score wiring at initialization, and EZNAS itself reports they saturate among top-decile candidates. Every sensible group assignment lives in that top decile. The random-weight result shows that untrained convolutions already exhibit translation invariance regardless of the data, so initialization-time scores have little reason to reflect data symmetry. The NASWOT and Synflow baselines and our own initialization ablation test this.
Why not just augment the data? On Galaxy10, the augmented standard network in the E(2) study reached 84.84% accuracy against 95.52% for the equivariant model, which also degraded far less under noise. Augmentation remains a cheap reference point, and we include it in the naive baseline's recipe.
Decisive Checks
The idea fails if the selected assignment varies across probe seeds more than EquiNAS assignments vary across search seeds, since LieGG's symmetry-variance finding makes this plausible. It also fails if the residual profile on the synthetic task does not locate the planted symmetry-breaking depth. It is supported if, on Galaxy10, the probe-selected network comes within seed noise of 20.3% test error at an order of magnitude less total compute, and if continuous-rotation layers chosen by the probe improve over the best discrete assignment.
References
- Equivariance-aware Architectural Optimization of Neural Networks
- Towards Architectural Optimization of Equivariant Neural Networks over Subgroups
- Group Equivariant Neural Architecture Search via Group Decomposition and Reinforcement Learning
- LieGG: Studying Learned Lie Group Generators
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- On Random Weights and Unsupervised Feature Learning
- Learning Lie Group Symmetry Transformations with Neural Networks
- E(2) Equivariant Neural Networks for Robust Galaxy Morphology Classification
- Rethinking Performance Estimation in Neural Architecture Search