Internal Representations of Vision Models Through the Lens of Frames on Data Manifolds

Henry Kvinge, Grayson Jorgenson, Davis Brown, Charles Godfrey, Tegan Emerson

Abstract

While the last five years have seen considerable progress in understanding the internal representations of deep learning models, many questions remain regarding the impact of model design choices on hidden representation geometry. In this work, we present a new approach to studying such representations inspired by the idea of a frame on the tangent bundle of a manifold. Our construction, the Neural Frame, is formed by assembling vectors representing specific types of perturbations—such as infinitesimal augmentations, noise, or generative model outputs—and tracking their evolution through a network. Our results provide new perspectives on how training with augmentation produces invariance and the trade-offs inherent in adversarial training.

Probing the Micro-Geometry of Neural Networks

Deep learning models are often described as projecting high-dimensional data onto a lower-dimensional "manifold." While researchers have tools to study the global topology of these manifolds, we have lacked precise methods to analyze local behavior. If a single image is nudged slightly—via a shift in hue or a rotation—how does the model’s internal representation change? Does the model collapse this change as irrelevant noise, or does it stretch it to distinguish features? Understanding this micro-geometry is crucial for explaining generalization and the costs of adversarial robustness.

The Neural Frame Framework

To bridge the gap between deep learning and differential geometry, we utilize the concept of a frame. In geometry, a $k$-frame is a set of $k$ linearly independent vectors in the tangent space of a point on a manifold.

Conceptual overview of Neural Frames: Augmentation frames (semantic changes) are preserved or expanded through layers, while noise frames are collapsed.

We define a Neural Frame by taking vectors representing specific perturbations and "pushing" them through the layers of a neural network. To measure the dimensionality of these frames robustly, we use Stable Rank—the ratio of the squared Frobenius norm to the squared spectral norm. This provides a measure of how many "dimensions" a frame occupies that is less sensitive to noise than standard algebraic rank. We primarily examine three frame types:

  • Augmentation Frames: Directions corresponding to semantic changes (e.g., rotation, brightness).
  • Noise Frames: Random Gaussian directions that likely point "off" the data manifold.
  • Diffusion Frames: Directions generated by diffusion models to simulate naturalistic variations.

Key Insights and Dynamics

Semantic Preservation vs. Noise Collapse

Models are remarkably adept at distinguishing "on-manifold" directions from "off-manifold" noise. As shown in the layer-wise analysis, random noise directions are rapidly collapsed, while augmentation directions are generally preserved or even expanded through the network.

Stable rank across layers for different frame types. Augmentation perturbations maintain dimensionality, whereas noise perturbations collapse rapidly after the first layer.

The Invariance Paradox

Common wisdom suggests that training with augmentations teaches a model to be invariant to those changes. However, Neural Frames reveal that the model actually preserves these variations intensely in its hidden layers. The "collapse" into a single invariant concept only occurs at the final classification layer. This suggests that internal representations remain highly sensitive to semantic nuances until the very last moment.

The Adversarial Trade-off

Adversarial training involves a known trade-off with standard accuracy. Our analysis shows that increasing the strength of adversarial training ($\epsilon$) causes the model to preserve noise directions more than a standard model would. Consequently, the model becomes less sensitive to semantically meaningful augmentation directions, providing a geometric explanation for the drop in standard performance.

Results and Conclusion

Our evaluation across architectures (CNNs vs. Transformers) and training regimes on ImageNet reveals several key trends:

  • Accuracy Correlation: There is a strong positive correlation between a model’s accuracy and its ability to preserve augmentation frames in its latent spaces.
  • Layer-wise Evolution: In a standard ResNet50, the stable rank for noise drops from ~17.5 at the input to near 1.0 by the output, while augmentation frames maintain a stable rank between 3.0 and 4.0.
  • Training Dynamics: Early in training, models locally compress the image manifold. As training progresses, they "unfold" the manifold, particularly in later layers, to better separate semantic features.

While limited by the use of software-defined augmentations as proxies for the true data manifold, the Neural Frame approach offers a powerful diagnostic tool. Future work may leverage these frames to design regularization techniques that explicitly guide the preservation or collapse of specific data directions.

Summary updated

Abstract

While the last five years have seen considerable progress in understanding the internal representations of deep learning models, many questions remain. This is especially true when trying to understand the impact of model design choices, such as model architecture or training algorithm, on hidden representation geometry and dynamics. In this work we present a new approach to studying such representations inspired by the idea of a frame on the tangent bundle of a manifold. Our construction, which we call a neural frame, is formed by assembling a set of vectors representing specific types of perturbations of a data point, for example infinitesimal augmentations, noise perturbations, or perturbations produced by a generative model, and studying how these change as they pass through a network. Using neural frames, we make observations about the way that models process, layer-by-layer, specific modes of variation within a small neighborhood of a datapoint. Our results provide new perspectives on a number of phenomena, such as the manner in which training with augmentation produces model invariance or the proposed trade-off between adversarial training and model generalization.
BibTeX
@inproceedings{kvinge2023internal,
  author = {Henry Kvinge and Grayson Jorgenson and Davis Brown and Charles Godfrey and Tegan Emerson},
  title = {{Internal Representations of Vision Models Through the Lens of Frames on Data Manifolds}},
  booktitle = {NeurReps @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=eY6zf3mk4d},
}
Figure