Abstract
While the last five years have seen considerable progress in understanding the internal representations of deep learning models, many questions remain regarding the impact of model design choices on hidden representation geometry. In this work, we present a new approach to studying such representations inspired by the idea of a frame on the tangent bundle of a manifold. Our construction, the Neural Frame, is formed by assembling vectors representing specific types of perturbations—such as infinitesimal augmentations, noise, or generative model outputs—and tracking their evolution through a network. Our results provide new perspectives on how training with augmentation produces invariance and the trade-offs inherent in adversarial training.
Probing the Micro-Geometry of Neural Networks
Deep learning models are often described as projecting high-dimensional data onto a lower-dimensional "manifold." While researchers have tools to study the global topology of these manifolds, we have lacked precise methods to analyze local behavior. If a single image is nudged slightly—via a shift in hue or a rotation—how does the model’s internal representation change? Does the model collapse this change as irrelevant noise, or does it stretch it to distinguish features? Understanding this micro-geometry is crucial for explaining generalization and the costs of adversarial robustness.
The Neural Frame Framework
To bridge the gap between deep learning and differential geometry, we utilize the concept of a frame. In geometry, a $k$-frame is a set of $k$ linearly independent vectors in the tangent space of a point on a manifold.
We define a Neural Frame by taking vectors representing specific perturbations and "pushing" them through the layers of a neural network. To measure the dimensionality of these frames robustly, we use Stable Rank—the ratio of the squared Frobenius norm to the squared spectral norm. This provides a measure of how many "dimensions" a frame occupies that is less sensitive to noise than standard algebraic rank. We primarily examine three frame types:
- Augmentation Frames: Directions corresponding to semantic changes (e.g., rotation, brightness).
- Noise Frames: Random Gaussian directions that likely point "off" the data manifold.
- Diffusion Frames: Directions generated by diffusion models to simulate naturalistic variations.
Key Insights and Dynamics
Semantic Preservation vs. Noise Collapse
Models are remarkably adept at distinguishing "on-manifold" directions from "off-manifold" noise. As shown in the layer-wise analysis, random noise directions are rapidly collapsed, while augmentation directions are generally preserved or even expanded through the network.
The Invariance Paradox
Common wisdom suggests that training with augmentations teaches a model to be invariant to those changes. However, Neural Frames reveal that the model actually preserves these variations intensely in its hidden layers. The "collapse" into a single invariant concept only occurs at the final classification layer. This suggests that internal representations remain highly sensitive to semantic nuances until the very last moment.
The Adversarial Trade-off
Adversarial training involves a known trade-off with standard accuracy. Our analysis shows that increasing the strength of adversarial training ($\epsilon$) causes the model to preserve noise directions more than a standard model would. Consequently, the model becomes less sensitive to semantically meaningful augmentation directions, providing a geometric explanation for the drop in standard performance.
Results and Conclusion
Our evaluation across architectures (CNNs vs. Transformers) and training regimes on ImageNet reveals several key trends:
- Accuracy Correlation: There is a strong positive correlation between a model’s accuracy and its ability to preserve augmentation frames in its latent spaces.
- Layer-wise Evolution: In a standard ResNet50, the stable rank for noise drops from ~17.5 at the input to near 1.0 by the output, while augmentation frames maintain a stable rank between 3.0 and 4.0.
- Training Dynamics: Early in training, models locally compress the image manifold. As training progresses, they "unfold" the manifold, particularly in later layers, to better separate semantic features.
While limited by the use of software-defined augmentations as proxies for the true data manifold, the Neural Frame approach offers a powerful diagnostic tool. Future work may leverage these frames to design regularization techniques that explicitly guide the preservation or collapse of specific data directions.
Summary updated