Neural frames: A Tool for Studying the Tangent Bundles Underlying Image Datasets and How Deep Learning Models Process Them

Henry Kvinge, Grayson Jorgenson, Davis Brown, Charles Godfrey, Tegan Emerson

Abstract

The manifold hypothesis suggests that high-dimensional data like images reside on low-dimensional manifolds. However, measuring these manifolds is challenging, limiting our understanding of deep learning dynamics. We introduce neural frames, a diagnostic tool inspired by differential geometry, to explore the local neighborhoods of data manifolds as they pass through neural network layers. By tracking how specific perturbations—such as rotations or noise—are processed, we reveal that models selectively preserve semantically meaningful variations while discarding noise. Our findings challenge the intuition that data augmentation leads to representational collapse, showing instead that it increases internal sensitivity to these variations.

Introduction: The Geometry of Vision

Despite the massive success of deep learning, models often remain "black boxes." While we can study global representation structures (e.g., how a whole dataset is clustered), we lack granular tools to understand local behavior. For instance, how does a model's internal geometry change when an image is slightly perturbed? Does it treat a minor rotation the same way it treats random pixel noise?

Understanding this local behavior is crucial for explaining generalization and the trade-offs in adversarial training. We address this gap by examining the "tangent bundle" of the data manifold—the space of all possible infinitesimal changes to an image—using a new framework called neural frames.

Methodology: The Neural $k$-Frame

A $k$-frame in differential geometry is a set of $k$ linearly independent vectors in the tangent space of a point on a manifold. We adapt this concept to neural networks through a three-step process:

  1. Frame Construction: We assemble vectors representing specific perturbations, such as Augmentation Frames (infinitesimal shifts in brightness, rotation, etc.) and Noise Frames (random Gaussian noise).
  2. The Push-forward: These vectors are passed through the layers of a neural network to observe how the internal representation of these local neighborhoods evolves.
  3. Measurement via Stable Rank: We use Stable Rank to quantify the "effective dimensionality" of these frames. A drop in stable rank indicates compression of those directions, while a high rank indicates preservation.
Conceptual diagram of the neural frame "push-forward" through network layers, showing how different types of perturbations (augmentation vs. noise) are mapped into the hidden representation space.

Key Insights and Results

Our analysis across various architectures (ResNets, ViTs, etc.) and training regimes on ImageNet yields several key findings:

Selective Compression

Models are highly discerning in how they process information. They rapidly collapse "off-manifold" directions (random noise) while preserving "on-manifold" directions (semantic augmentations) throughout most of the network. As shown in the figure below, the stable rank of noise frames drops precipitously after the first layer, whereas augmentation frames maintain their dimensionality until the final output.

Stable rank across layers for different frame types in a ResNet50. Noise frames collapse immediately, while augmentation and diffusion-based frames maintain higher effective dimensionality until the final layers.

The Invariance Paradox

A common intuition is that data augmentation makes a model invariant by "collapsing" those variations. Our results show the opposite: models trained with heavy augmentation actually preserve those modes of variation more faithfully in their hidden layers. Invariance is only achieved at the very final classification layer.

Adversarial Trade-offs

Adversarial training introduces a geometric shift. As the robustness parameter ($\epsilon$) increases, models begin to preserve noise directions more and semantic augmentation directions less. This provides a geometric explanation for why adversarial robustness often degrades standard accuracy: the model's internal geometry becomes less aligned with the natural data manifold.

Conclusion

Neural frames provide a powerful window into the inner workings of deep learning models. By demonstrating that high-performing models better "fit" the underlying data manifold and selectively preserve semantic variations, this tool offers a path toward more interpretable and geometrically-grounded architecture design.

Summary updated

Abstract

The assumption that many forms of high-dimensional data, such as images, actually live on low-dimensional manifolds, sometimes known as the manifold hypothesis, underlies much of our intuition for how and why deep learning works. Despite the central role that they play in our intuition, data manifolds are surprisingly hard to measure in the case of high-dimensional, sparsely sampled image datasets. This is particularly frustrating since the capability to measure data manifolds would provide a revealing window into the inner workings and dynamics of deep learning models. Motivated by this, we introduce neural frames, a novel and easy to use tool inspired by the notion of a frame from differential geometry. Neural frames can be used to explore the local neighborhoods of data manifolds as they pass through the hidden layers of neural networks even when one only has a single datapoint available. We present a mathematical framework for neural frames and explore some of their properties. We then use them to make a range of observations about how modern model architectures and training routines, such as heavy augmentation and adversarial training, affect the local behavior of a model.
BibTeX
@inproceedings{kvinge2023neural,
  author = {Henry Kvinge and Grayson Jorgenson and Davis Brown and Charles Godfrey and Tegan Emerson},
  title = {{Neural frames: A Tool for Studying the Tangent Bundles Underlying Image Datasets and How Deep Learning Models Process Them}},
  booktitle = {NeurReps @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=eY6zf3mk4d},
}
Figure