Symmetry-Safe 2:4 Sparsity for Rotation-Equivariant Vision Transformers

Vision transformers that are built to respect rotations and reflections are more accurate than ordinary ones on data with no preferred orientation, such as pathology slides, satellite tiles, or rotated digits. They are also slow. The usual construction makes one copy of every feature per rotation or reflection, so an 8-element symmetry group makes activations 8 times larger. Pruning, which means setting most weights to zero, looks like a natural way to pay that cost back, but it runs into two problems at once. Zeroing individual weights destroys the symmetry guarantee. Scattered zeros also rarely make GPUs faster, because GPUs are built for dense, regularly laid-out arithmetic.

This note proposes a fix for both problems. If the weights of an equivariant layer are rewritten in the right coordinate system, the symmetry constraint becomes purely a block pattern, and the entries inside each block are unconstrained. Any zero pattern inside those blocks is then symmetry-safe. That includes the 2:4 pattern (two nonzeros in every four consecutive weights) that NVIDIA's sparse tensor cores execute at double speed. The two savings multiply. The block structure removes most of the group overhead, the hardware sparsity halves what is left, and the network stays exactly equivariant. Here equivariant means that rotating the input rotates the internal features in a known way.

Core Claim

Consider a layer mapping $c$ copies of the regular representation of a finite group $G$ to itself. Stated plainly, that is $c$ channels, each carrying one feature per group element. When the layer is written in the group's Fourier basis, meaning its irreducible representations ("irreps": the smallest pieces the group action splits into), it takes the form $\bigoplus_\rho (W_\rho \otimes I_{d_\rho})$. Here $\rho$ ranges over the irreps, $d_\rho$ is the dimension of irrep $\rho$, and $W_\rho$ is a free real matrix of size $c d_\rho \times c d_\rho$ whenever $\rho$ is of real type. All irreps of the dihedral groups $D_4$ (p4m) and $D_8$ are of real type.

Each block runs as one ordinary matrix multiply with $W_\rho$, applied to activations reshaped so that the $d_\rho$ copies become extra columns. The multiply-add count per token drops from the regular-basis cost to the irrep-basis cost:

$$ c^2|G|^2 \;\longrightarrow\; c^2 \sum_\rho d_\rho^3 . $$

The formula follows from the block sizes above. For $D_4$ the ratio is 64 to 12. For $D_8$ it is 256 to 28.

The central claim is that equivariance places no constraint on the entries of $W_\rho$. A 2:4 mask learned independently inside each $W_\rho$ therefore preserves exact equivariance by construction. It adds a hardware-executed 2x on top of the block savings, with no custom unstructured kernel.

This addresses a tension that the prior literature leaves unresolved. The equivariant strong lottery ticket work (A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis) shows that symmetry-safe pruning must remove coefficients of an equivariant basis rather than raw matrix entries. It prunes those coefficients without structure and never measures runtime. When such coefficients are mapped back to a dense weight matrix, the resulting zero pattern is irregular. The sparse-kernel literature says that irregular pattern is the wrong shape for GPUs. Sputnik (Sparse GPU Kernels for Deep Learning) only beats dense cuBLAS above 71% sparsity, and SMaT (High Performance Unstructured SpMM Computation Using Tensor Cores) only above 78%. The irrep basis is the one coordinate system in which "symmetry-safe" and "hardware-shaped" coincide.

Relation to Prior Work

The proposal generalizes Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency. That paper block-diagonalizes linear layers for horizontal flips only (a two-element group), halving linear-layer FLOPs and gaining nearly 60% throughput on an A100 for ResMLP-L24. It notes that larger groups with two-dimensional irreps complicate the construction, and that on ViT-S, unfused handling of split feature maps makes the model slower than a dense one.

We take that paper's caution about larger groups as a statement about execution rather than arithmetic. For dihedral groups the arithmetic savings actually grow with group size. The real difficulties are two. First, pointwise nonlinearities such as GELU are equivariant only in the regular basis, not the irrep basis. Second, the $I_{d_\rho}$ Kronecker factor forces an activation reshape. Both must be fused into the kernel epilogue, or the small-model slowdown the paper observed will reappear. We also add hardware sparsity, which that paper does not touch.

The architectural starting point is REViT: Roto-reflection Equivariant Convolutional Vision Transformer. REViT lifts inputs to the group and produces queries, keys, and values with group convolutions. Attention scores are dot products over the group fiber, and these are unchanged by an orthogonal change of basis, so the Fourier-basis rewrite leaves attention itself untouched. Only the query/key/value projections and the per-token feed-forward layers, where most of the FLOPs sit, are restructured. $E(2)$-Equivariant Vision Transformer and Attentive Group Equivariant Convolutional Networks motivate the work by documenting the cost of group lifting. The latter reports 72GB of CUDA memory against 5GB for the standard All-CNN on CIFAR-10.

The sparse-training methods supply the tools. SR-STE from Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch learns 2:4 masks from scratch without mask flicker. ARMOR (ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization) is the post-training alternative. Its block-diagonal corrector matrices happen to commute with our block structure if they are applied per irrep, which makes it a natural extension.

E2Former-V2: On-the-Fly Equivariant Attention with Linear Activation Memory is the closest precedent in spirit. It uses a basis change from 3D to 2D rotations to turn dense equivariant tensor products into cheap structured operations, then fuses them into a Triton kernel. It works on continuous 3D groups in FP32 and uses no weight sparsity. We transfer its lesson, "change basis first, then fuse", to discrete 2D groups and half-precision sparse tensor cores.

Research Program

The first component is the layer itself. For p4m and p8m, we implement Fourier-basis projection and feed-forward layers in which each irrep block is a separate half-precision matrix multiply. Each block carries its own 2:4 mask trained with SR-STE.

Two open design questions follow from this layer. The first is how to handle nonlinearities. One option applies an inverse group Fourier transform, then GELU, then the forward transform, fused into the multiply epilogue. This costs $O(c|G|^2)$ per token, which is not negligible at small $c$ and must be measured. The other option uses gated nonlinearities that act directly on irrep norms, as in steerable CNNs. These are cheaper but possibly less accurate.

The second question is how to allocate sparsity across irreps. Low-frequency and high-frequency blocks may differ in how much pruning they tolerate, analogous to the per-layer ERK allocations in Rigging the Lottery: Making All Tickets Winners. We will test uniform 2:4, 4:8, and a per-irrep choice between dense and 2:4 selected by a small search on validation data.

We will also verify equivariance numerically at every stage, since the claim is exact rather than approximate.

The second component is the kernel. We write a Triton or CUTLASS kernel that batches all irrep blocks of one layer into a grouped sparse matrix multiply. It fuses the reshape for the $d_\rho$ copies and the nonlinearity epilogue so that intermediate lifted tensors never travel to main GPU memory. The analysis side is a simple cost model that predicts throughput from $c$, $G$, and the per-irrep sparsity pattern, checked against measurements. This tells us where the method stops paying off, for example at small $c$, where transform overhead dominates.

Baselines

The naive reference point is a standard dense ViT-Small trained with rotation and flip augmentation. It shows what equivariance buys at all. REViT reports 72.08% ImageNet top-1 for vanilla ViT-Small.

The strongest existing method is dense p4m REViT. REViT: Roto-reflection Equivariant Convolutional Vision Transformer reports 79.27% ImageNet-1K top-1 accuracy and 98.92% on Rotated MNIST with its p8 variant. No reported result on Lacuna was found for REViT's wall-clock throughput, so we will measure it ourselves on the same A100 and at the same batch size and precision. We will retrain it under our exact recipe rather than quote the number. Beating it means staying within 0.5 points of its top-1 accuracy at a measured end-to-end throughput gain of at least 2x. The block rewrite is expected to deliver the larger share of that gain and 2:4 the rest.

The first ablation is the same Fourier-basis model kept dense. It isolates the contribution of 2:4 sparsity.

The second ablation isolates the key ingredient, sparsity in the irrep basis. It is orbit-tied 2:4 in the regular basis: masks are shared across each group orbit and grouped along the channel axis, which also preserves equivariance and also runs on sparse tensor cores. It is given the same SR-STE training budget.

The third comparison uses unstructured coefficient pruning in the style of the equivariant lottery ticket paper, at 80% and 90% sparsity, executed with Sputnik and SMaT. It represents the unstructured-sparsity route.

The fourth comparison is a width-reduced dense Fourier-basis model whose FLOPs match the sparse one. It tests whether sparsity beats simply using fewer channels.

All models share the data pipeline, number of epochs, optimizer, and a matched hyperparameter search of equal trial count.

For hardware sparsity on standard networks, the only reference accuracy cost in the file is ResNet-50 with 2:4, at 77.0% against a 77.3% dense baseline (Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch). A drop much larger than roughly 0.3 points for the equivariant model would suggest that the irrep blocks are harder to sparsify than ordinary weights.

Why not just…?

Why not just train a dense ViT with rotation augmentation? On ImageNet-1K, REViT's equivariant model beats augmented ViT-Small by about seven points (79.27% vs 72.08%). Identifiable Equivariant Networks are Layerwise Equivariant finds that augmentation yields only approximate internal symmetry. Augmentation also saves no compute. Our dense baseline reproduces this comparison under matched settings, so the accuracy gap is measured rather than assumed.

Why not just prune the equivariant model without structure and run it with Sputnik or SMaT? Raw-entry pruning breaks equivariance, according to the equivariant lottery ticket paper. Symmetry-safe coefficient pruning produces irregular patterns that give speedups only above the 71–78% sparsity thresholds reported for those kernels. SMaT's row-reordering preprocessing also rules it out for masks that change during training. Our third comparison measures accuracy and speed at 80% and 90% sparsity directly. If an equivariant ViT tolerates 90% unstructured sparsity with little loss, this route could compete at inference, and we will report that honestly.

Why not just apply orbit-tied 2:4 in the ordinary regular basis? This is the real threat, because it is simpler and also exactly equivariant. It gets the 2x from the hardware but keeps the $c^2|G|^2$ cost of materializing the block-circulant weight matrix. The irrep basis removes a further 5.3x for $D_4$ at the arithmetic level. The open question is whether that arithmetic advantage survives the transform and reshape overheads, especially at small width. The second ablation settles this. If the regular-basis version matches our throughput at ViT-S scale, the contribution narrows to larger groups and wider models, and the note will say so.

Why not just make the dense model narrower? Fewer channels also halve FLOPs, need no sparse hardware, and are therefore portable beyond Ampere and Hopper GPUs. Sparse training results such as RigL's suggest that sparse wide networks beat dense narrow ones at equal FLOPs. That has not been shown for equivariant transformers, which is why the width-matched model is a required baseline.

Proposed Experiments

We train p4m and p8m models on Rotated MNIST, PatchCamelyon, CIFAR-10, and ImageNet-1K at ViT-Small scale. We report top-1 accuracy, equivariance error (maximum output difference under all group actions, which should sit at half-precision rounding level for p4m), peak memory, and measured images per second on an A100. PatchCamelyon references are GE-ViT's 82.73% ($E(2)$-Equivariant Vision Transformer) and the 10.88% error of attentive p4m (Attentive Group Equivariant Convolutional Networks).

Four checks decide the outcome. The first is equivariance error: zero for our method and for orbit-tied 2:4, and clearly nonzero for raw-entry pruning. The second is accuracy within half a point of dense REViT under 2:4. The third is end-to-end throughput above both REViT and the orbit-tied regular-basis model, measured as a function of width $c$ to locate where the overhead of the Fourier transform for the nonlinearity is repaid. The fourth is a win over the FLOP-matched narrow dense model.

For p8m, interpolation artifacts from 45-degree rotations on the pixel grid limit input-level equivariance, as REViT and GE-ViT both note. We will therefore report internal-layer equivariance separately from end-to-end equivariance.