Low-Rank Symmetry Relaxation: Getting the Training Benefit of Broken Equivariance at a Fraction of the Cost

Equivariant networks are built so that rotating the input rotates the output in a matching way. They learn efficiently when data really is symmetric. Recent work has found something surprising: letting each layer temporarily break that symmetry during training helps the network find better solutions, even when the task is perfectly symmetric. Afterward the symmetry-breaking parts shrink to zero and can be deleted. The catch is cost. Current methods attach a full, unconstrained copy of each layer next to the equivariant one. Recurrent Equivariant Constraint Modulation (RECM) reports roughly 40% longer training for this, and ACE stores both parameter sets in every layer throughout training.

We propose to test a simple hypothesis. The directions in which a layer usefully breaks symmetry are few. If so, a small low-rank, non-equivariant correction per layer, gated by one learned scalar, should recover most of the accuracy and optimization benefit of full relaxation. Its training overhead would scale with a chosen rank $r$ rather than with layer width. On symmetric tasks the correction is removed after training. On tasks with broken symmetry, only a thin correction survives in the layers that need it. If the hypothesis is false, and relaxation needs full-rank freedom, that is also worth knowing, because it would mean the 40% overhead is intrinsic rather than an artifact of design.

Core Claim

Consider layer $i$ of an equivariant network, with input feature $x$ flattened per point or pixel into a vector of size $d$ (for vector-valued features, the geometric axis is included). We replace the full homotopic layer of ACE, $f_{eq}^{(i)}(x) + \gamma_i f_{neq}^{(i)}(x)$, with

$$f^{(i)}(x) = f_{eq}^{(i)}(x) + \gamma_i\, U_i V_i^{\top} x,$$

In this layer, $U_i, V_i \in \mathbb{R}^{d \times r}$ are unconstrained matrices. Because they mix across the geometric axis, they break equivariance. $\gamma_i$ is a per-layer scalar gate, and $r \ll d$.

The added cost per point is about $2dr$ multiply-adds, instead of the $d^2$ of a dense unconstrained branch. The gate is controlled exactly as in ACE: an equality constraint pushes $\gamma_i$ to zero on symmetric tasks, and a slack-relaxed inequality constraint lets it stay nonzero where symmetry is broken. Each layer then needs only one extra scalar Lagrange multiplier, so the dual bookkeeping is negligible. The costly part of prior methods is the branch, not the multiplier.

The new claim is empirical and falsifiable. At small $r$ (we expect 4 to 16), the low-rank version matches full-branch RECM and ACE on their own benchmarks while cutting most of their training-time overhead.

Relation to Prior Work

Work that supplies the gate and the pruning logic. RECM and ACE both show that training-time relaxation can be pruned to a strictly equivariant network. RECM reports 0% inference parameter overhead on symmetric tasks and a +1.25 parameter ratio on aligned ModelNet. We keep ACE's constraint-based gate. We do not use RECM's recurrent symmetry-gap state, which adds its own decay hyperparameter. The one thing we change is the capacity of the relaxation branch, which neither paper varies.

Work that motivates the need for per-layer, data-dependent relaxation. Several papers relax symmetry through other mechanisms. Partial G-CNNs learn subsets of the symmetry group to sample from. Probabilistic partial steerable CNNs learn a distribution over group elements. Marginal-likelihood layer-wise equivariance chooses the degree of symmetry by Bayesian model selection, using second-order curvature estimates. EquiNAS architecture search searches over which layers are equivariant. Each confirms that the best amount of symmetry varies by layer and by task. Each also pays for this through sampling, curvature estimates, or search. We aim for the cheapest mechanism that still lets the right amount of symmetry vary per layer.

Work that frames the question theoretically. Identifiable Equivariant Networks are Layerwise Equivariant proves that end-to-end equivariance forces layer-wise equivariance in identifiable networks. It says nothing about how gradient descent reaches such solutions. Our rank sweep gives a first measurement of how many extra dimensions optimization needs in order to get there.

Work that frames the cost trade-off. The training-only scaffolding literature (REPA, BatchFormer) accepts extra training compute in exchange for a clean model at deployment. Our position is that, for symmetry relaxation, most of that training compute may be unnecessary.

Research Program

Diagnostic step, before building anything. We train full-branch ACE and RECM models on rotated ModelNet40 and the N-body benchmark, using the published setups. At several checkpoints we measure the singular value spectrum of each layer's unconstrained contribution. Because the branch acts through $\gamma_i f_{neq}^{(i)}$, we measure the effective rank of the product and also of the change in weights since the previous checkpoint. If most of the energy lies in a few directions throughout training, the hypothesis is plausible before the new method is written.

Main study. We implement the low-rank layer on the same backbones: Vector Neuron PointNet for ModelNet40 and SEGNN for N-body. We sweep $r$ over {1, 2, 4, 8, 16, 32, full}. For each run we record:

  • accuracy or MSE,
  • wall-clock training time to a target metric,
  • peak training memory,
  • inference parameters after pruning.

Mechanism study. We track $\gamma_i$ and the equivariance error per layer over training. The goal is to check that the low-rank version reproduces the pattern prior work reports: early relaxation that later collapses on symmetric tasks, and persistent relaxation in deeper layers on asymmetric tasks.

Baselines

Naive reference. The strictly equivariant backbone with no relaxation: VN-PointNet on ModelNet40 and SEGNN on N-body. We run it under the same wall-clock budget as the slowest relaxed method. Concretely, it gets about 40% more steps, with learning rate and schedule tuned on the same grid. Without this, any gain from relaxation could be a gain from compute. For reference, the file reports VN-PointNet at 0.68 accuracy on rotated ModelNet40 and SEGNN at MSE 5.6 on N-body, both taken from the RECM paper.

Strongest existing method. Full-branch RECM, which reports 0.80 accuracy on rotated ModelNet40 and MSE 3.7 on N-body, at about 40% extra training time (RECM). We also run full-branch ACE, which reports a +2.78 percentage-point ModelNet40 gain over baselines (ACE). Both are re-run with their published hyperparameters on our hardware, so that training-time measurements are directly comparable.

What counts as success. Low rank "beats" these methods if it reaches their accuracy or MSE within seed noise, over at least 3 seeds, while cutting their measured training-time overhead by at least half.

Ablations isolating the key ingredient. Three ablations test what the low-rank correction is actually doing:

  • Learned versus noise. A rank-$r$ branch with frozen random $U_i, V_i$ and a learned gate tests whether learned directions matter, or whether any symmetry-breaking perturbation helps.
  • Per-layer versus global. A single global gate shared across layers tests whether per-layer control is needed.
  • Rank versus parameters. A full-rank branch with its width reduced to match the low-rank branch's parameter count separates the effect of rank from the effect of parameter count.

Why not just…?

Why not just train the strict equivariant model longer? This is the most serious threat. RECM's gains might partly reflect extra compute and parameters rather than relaxation. The compute-matched naive baseline above tests this directly. If the longer-trained strict model closes the gap to RECM, the story changes: the finding becomes that relaxation mostly buys nothing on symmetric tasks, and we would report that. Evidence against this outcome already exists. ACE reports that strictly equivariant models failed to converge at 85% point removal, where relaxed models remained stable, which suggests an optimization barrier that more steps alone may not cross.

Why not just use an unconstrained model with heavy data augmentation? It has no training-time relaxation overhead at all. Prior results point the other way, however. ACE matches a standard equivariant network's N-body MSE with 44% less data. Model-compression studies found that strong augmentation cost small models about 2% accuracy on CIFAR-100. We include an augmented unconstrained PointNet-style model of matched size on rotated ModelNet40 as a reference point, not as the main competitor.

Why not just anneal a single global relaxation coefficient, or inject noise, instead of learning directions? Either would be simpler still. If the frozen-random low-rank ablation or the global-gate ablation matches the learned per-layer version, then the benefit comes from generic perturbation, not from learned symmetry breaking. That would make the method even cheaper, and we would recommend it. Prior work showing that per-layer symmetry profiles vary by task (probabilistic steerable CNNs) predicts that per-layer gates will matter on asymmetric tasks. We expect that difference to show up on aligned ModelNet40 and Galaxy10, but not necessarily on symmetric tasks.

Why not distill from a relaxed teacher into a strict student? This moves the relaxation outside the network. It still requires training the expensive relaxed teacher first, so total compute rises rather than falls. It is out of scope here.

Decisive Experiments

Rotated ModelNet40. The central test is rotated ModelNet40. If rank 8 reaches about 0.80 accuracy at well under RECM's training overhead, the hypothesis holds. If accuracy falls toward 0.68 below full rank, relaxation needs full capacity, and the prior overhead is justified.

N-body. The same test is repeated on N-body against MSE 3.7.

Tasks with genuinely broken symmetry. We use two further tasks. On aligned ModelNet40, the question is whether the pruned low-rank model beats RECM's +1.25 inference parameter ratio at similar accuracy. On Galaxy10 DECals, the targets are a uniform $D_4$ baseline at 50.8% test error and EquiNAS at 20.3% (EquiNAS). The question there is whether a thin correction is enough when symmetry must be broken permanently.

Sanity check. MNIST6-180 checks that even $r=1$ can separate 6 from 9. The Partial G-CNN paper reports 50% for a strict G-CNN versus 100% for its partial model.

Diagnostic cross-check. The effective-rank measurement on trained full-branch models should predict the smallest rank that works in each case. If it does, we have both a cheap method and an explanation of why it works.