Internal Representations of Vision Models Through the Lens of Frames on Data Manifolds

Henry Kvinge, Grayson Jorgenson, Davis Brown, Charles Godfrey, Tegan Emerson

TL;DR

Researchers have introduced neural frames, a new diagnostic tool inspired by differential geometry to peek inside the "black box" of vision models. By tracking how a model transforms a set of local perturbations (like tiny rotations or noise) layer-by-layer, they discovered that robust models actually preserve meaningful variations longer than previously thought, while rapidly collapsing random noise.

What Problem Does This Paper Address?

We know deep neural networks are incredibly good at distilling raw pixels into semantic concepts, but the "mechanics" of this transformation remain elusive. Most existing tools for studying internal representations focus on the "big picture"—the global topology or mutual information of an entire dataset.

However, these methods often miss what happens at the local level: how does a model behave in the immediate neighborhood of a single data point? For instance, if you slightly shift the hue of an image, does the model "ignore" it immediately to achieve invariance, or does it carefully process that change? Understanding these local dynamics is crucial for explaining phenomena like adversarial vulnerability and the impact of data augmentation.

Methods

The authors propose the neural frame, a construction grounded in the theory of manifolds.

  1. The Frame: In geometry, a k-frame is a set of $k$ linearly independent vectors in the tangent space of a manifold. In this paper, a frame is built by taking a single image and creating several "infinitesimal" perturbations of it—such as tiny rotations, brightness shifts, or Gaussian noise.
  2. Layer-by-Layer Tracking: They pass these perturbed images through a network and calculate the differential (the Jacobian) to see how those initial vectors are deformed, compressed, or expanded at each layer.
  3. Stable Rank: To quantify this, they use stable rank—a noise-robust version of matrix rank. A high stable rank means the model is preserving the distinct directions of variation; a low stable rank means the model is collapsing those directions together.
  4. Frame CKA: They also adapt Centered Kernel Alignment (CKA) to compare how different layers or models represent the exact same local frame, providing a more granular view than standard CKA.

Datasets

The study primarily utilizes ImageNet. They evaluate a wide variety of pre-trained architectures, including:

  • CNNs: ResNet50, ResNet18, ResNeXT50, DenseNet121, InceptionV3, and AlexNet.
  • Vision Transformers (ViTs): ViT-Base and Swin Transformer.
  • Robust Models: ResNet50s trained with adversarial perturbations ($\ell_2$-robust), Deep Augmentation, PRIME, and Stylized ImageNet.

Key Insights and Ideas

  • Semantic Preservation vs. Noise Collapse: Models are remarkably good at distinguishing between "natural" variations and random noise. As shown in the cartoon below, models tend to preserve "on-manifold" augmentation directions while aggressively collapsing "off-manifold" noise directions in the very first layers.
Figure 1: A cartoon visualizing an augmentation frame with three tangent vectors (derived from hue shift, brightness shift, and a small rotation and crop) and a noise frame with three noise vectors. We think of the augmentation vectors as approximately living in the tangent space of point x on the data manifold while the noise vectors do not. Our cartoon illustrates that models process different frames differently with the noise vectors in this cartoon being collapsed by the first ℓlayers of the network.
  • The Invariance Paradox: Intuitively, one might think that training with augmentations makes a model "blind" to those changes (invariance). Instead, the authors found that models trained with heavy augmentation actually preserve those variations more faithfully in their hidden layers, only collapsing them at the final classification stage.
  • Adversarial Trade-offs: Adversarial training (increasing $\epsilon$) forces the model to increasingly preserve noise directions at the direct expense of semantically meaningful augmentation directions. This provides a geometric perspective on why adversarial robustness often hurts standard generalization.
  • Accuracy Correlation: There is a positive correlation between a model's accuracy and its ability to preserve augmentation directions (measured by stable rank).

Key Results

  • Layer Dynamics: In standard ResNets, the stable rank for noise drops from ~18 to ~2 in the first layer, whereas the stable rank for augmentations stays high (around 4) until the very end of the network.
Figure 2: The stable rank of different types of frames measured at various layers of a ResNet50 model. Layer zero corresponds to model input and Layer 6 corresponds to model output. Shaded regions indicate 95% confidence intervals over 40 randomly selected ImageNet images.
  • Augmentation Impact: Models trained with "Deep Augmentation" or "PRIME" show significantly higher stable rank for augmentation frames across all hidden layers compared to standard training.
Figure 3: The stable rank (by layer) of ResNet50 models evaluated with respect to augmentation frames on ImageNet images where each model was trained with a different augmentation method. Layer zero corresponds to model input and the last layer corresponds to model output. Shaded regions indicate 95% confidence intervals over 40 randomly selected ImageNet images.
  • Adversarial Robustness: As the adversarial constraint $\epsilon$ increases, the "Augmentation Frame" stable rank decreases, while the "Noise Frame" stable rank actually increases in early layers, suggesting the model is becoming more "attuned" to noise-like directions.
Figure 4: (Left) The stable rank (by layer) of adversarially trained l2-robust ResNet50 models with varying ϵ values evaluated with respect to augmentation frames on ImageNet images. Layer zero corresponds to model input and the last layer corresponds to model output. (Right) The same models evaluated on noise frames. Shaded regions indicate 95% confidence intervals over 40 randomly selected ImageNet images.

Limitations and Future Work

The authors acknowledge that their chosen frames only span a proper subspace of the full tangent space; there may be important directions of variation that their specific augmentations don't capture. Furthermore, the "data manifold" is a theoretical construct; in practice, digital images are discrete, which can lead to "edge effects" during transformations like rotation (though the authors propose a resizing/cropping workaround for this).

Future work could involve finding "optimal" frames that capture the most relevant information for a specific task or using these tools to design architectures that better mimic the manifold-preserving properties of robust models.

Summary updated

Abstract

While the last five years have seen considerable progress in understanding the internal representations of deep learning models, many questions remain. This is especially true when trying to understand the impact of model design choices, such as model architecture or training algorithm, on hidden representation geometry and dynamics. In this work we present a new approach to studying such representations inspired by the idea of a frame on the tangent bundle of a manifold. Our construction, which we call a neural frame, is formed by assembling a set of vectors representing specific types of perturbations of a data point, for example infinitesimal augmentations, noise perturbations, or perturbations produced by a generative model, and studying how these change as they pass through a network. Using neural frames, we make observations about the way that models process, layer-by-layer, specific modes of variation within a small neighborhood of a datapoint. Our results provide new perspectives on a number of phenomena, such as the manner in which training with augmentation produces model invariance or the proposed trade-off between adversarial training and model generalization.
BibTeX
@inproceedings{kvinge2023internal,
  author = {Henry Kvinge and Grayson Jorgenson and Davis Brown and Charles Godfrey and Tegan Emerson},
  title = {{Internal Representations of Vision Models Through the Lens of Frames on Data Manifolds}},
  booktitle = {NeurReps @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=eY6zf3mk4d},
}
Figure