TL;DR
Researchers have introduced SEIS (Subspace-based Equivariance and Invariance Scores), a new metric that "looks under the hood" of neural networks to see how they handle geometric changes like rotations or shifts. Unlike previous methods that only look at final predictions, SEIS can distinguish between a layer that has completely lost spatial information and one that has simply re-encoded it into a different internal format.
What Problem Does This Paper Address?
Deep learning models, especially Convolutional Neural Networks (CNNs), are designed with the assumption that they should handle spatial transformations gracefully. We generally want two things:
- Equivariance: If an input moves, the internal representation should move accordingly (preserving the "where").
- Invariance: The final classification should remain the same regardless of the movement (the "what").
The problem is that current tools for measuring these properties are limited. Most methods look at "prediction stability"—checking if the final output changes when the input is rotated. However, this is a "black box" approach. It can't tell you where in the network the spatial information was lost or if a middle layer is still tracking the object's position but using a different internal "basis" to do so. Existing representational similarity tools (like CKA or SVCCA) are usually used to compare different models, not to analyze how a single model responds to geometric shifts.
Methods
The authors propose SEIS, which adapts Singular Vector Canonical Correlation Analysis (SVCCA) using a novel spatially-aware matricization strategy.
- The "Flip": Standard similarity metrics treat spatial locations (pixels) as independent samples. SEIS flips this: it treats spatial coordinates as features and aggregates channel responses across the batch as observations. This allows the metric to measure the stability of the spatial basis itself.
- Denoising: They use Singular Value Decomposition (SVD) to keep only the top components (explaining 99% of variance), filtering out noise and redundant neurons.
- Two Scores:
- $S_{equiv}$ (Equivariance): Measures if the information is preserved up to a linear transformation. A high score means the "info is still there," even if it's been moved or permuted.
- $S_{inv}$ (Invariance): Measures if the spatial feature basis remains aligned. A high score means the representation didn't move at all despite the input changing.
Datasets
The researchers validated SEIS across several standard benchmarks:
- MNIST: Used for synthetic validation where transformations (rotation, scaling, translation) were manually applied to activations to prove the metric works.
- CIFAR-100: Used to train ResNet-18 architectures to study how these properties evolve during training and across different depths.
- PASCAL VOC 2012: Used for complex tasks like semantic segmentation and multi-task learning (MTL) to see how different objectives shape internal geometry.
Key Insights and Ideas
- The "Where-to-What" Pipeline: In ResNets, there is a clear trajectory. Early layers establish strict equivariance (tracking the "where"), which is gradually traded off for invariance (the "what") in deeper layers.
- Data Augmentation is a Double Win: Training with data augmentation doesn't just make the final output more robust; it actually increases invariance in deep layers while simultaneously preserving equivariance better than non-augmented models.
- Multi-Task Synergy: One of the most surprising findings was that Multi-Task Learning (MTL)—training on classification and segmentation at once—creates a "synergistic gain." The shared encoder becomes better at both equivariance and invariance compared to models trained on just one task.
- Skip Connections as "Geometric Bypasses": In U-Net architectures, the authors found that equivariance is often lost during the "bottleneck" of decoding but is actively restored by skip connections, which re-inject spatial fidelity from earlier layers.
Key Results
- Validation: On synthetic tests, SEIS correctly assigned scores near 1.0 for perfect identity and high equivariance scores (>0.85) for transformations, while random baselines scored near zero.
- Depth Evolution: In a ResNet-18, equivariance starts high (~0.9) and drops as you go deeper, while invariance starts near 0.1 and climbs to ~0.3 in the final layers.
- MTL Performance: The Multi-Task model achieved the highest scores for both metrics at the bottleneck ($S_{inv} = 0.22$, $S_{equiv} = 0.54$), outperforming single-task classification or segmentation models.
Limitations and Future Work
While SEIS provides a powerful lens, the authors note (or the methodology implies) a few areas for growth:
- Linear Assumption: SEIS relies on CCA, which looks for linear relationships. While the authors argue that modern architectures preserve local topology linearly, highly non-linear re-encodings might still be "invisible" to this metric.
- Computational Overhead: Performing SVD and CCA across many layers and large batches can be computationally intensive compared to simpler metrics.
- Beyond Grids: The current matricization strategy is designed for grid-structured data (images). Extending this to non-grid data, like point clouds or graphs, remains a future challenge.
Summary updated