Representational Geometry as a Diagnostic for Adversarial Robustness

Brandon Ismalej, Alex Cloninger, Gal MishnearXiv 2026 · arXiv

TL;DR

This paper argues that standard accuracy metrics are insufficient for auditing adversarial robustness, as they fail to capture how a network's internal representation of data changes under attack. By using geometric analysis and dimensionality reduction (M-PHATE), the authors demonstrate that defenses which appear most robust by accuracy can actually be the most geometrically unstable, suggesting that representational geometry should be used as a primary diagnostic tool.

What Problem Does This Paper Address?

In the world of deep learning, "robustness" is the ability of a model to resist adversarial attacks—small, intentional perturbations designed to cause misclassification. Currently, the industry standard for measuring robustness is a single scalar: accuracy under attack.

However, the authors argue this is a "black box" metric. Two networks might achieve identical robust accuracy while organizing their internal data representations in completely different ways. Accuracy only tells us if a decision boundary was crossed; it doesn't tell us how much the internal "logic" of the network drifted or how the training process reshaped the data's geometry. Without understanding these structural changes, we lack a complete picture of why a defense works—or when it might fail.

Methods

The researchers moved beyond simple accuracy by treating neural network activations as geometric objects. They focused on the penultimate layer (the last layer before the final classification) of Multilayer Perceptrons (MLPs).

To visualize and quantify these high-dimensional changes, they employed several key techniques:

  • M-PHATE: An extension of the PHATE dimensionality reduction method. It allows researchers to visualize "trajectories"—how the representation of data samples evolves epoch-by-epoch during training—and "joint embeddings" to see how clean and perturbed samples sit relative to each other in the same space.
  • Geometric Metrics:
    • Normalized Displacement ($d_{norm}$): Measures how far an adversarial perturbation pushes a data point in the latent space, relative to the typical distance between clean points.
  • Maximum Mean Discrepancy (MMD): A statistical test to determine if the distribution of clean points and perturbed points are significantly different.
  • k-Nearest Neighbor (kNN) Recovery: Measures if a perturbed point still stays close to its original "neighbors" in the latent space.

Datasets

The study utilized the MNIST dataset of handwritten digits. While a relatively simple dataset, it provided a controlled environment to establish a proof-of-concept for geometric diagnostics. The authors used a fixed batch of 1,000 training images and 2,000 test images across all experiments to ensure consistency.

Key Insights and Ideas

  • The Accuracy-Geometry Paradox: The most striking finding is that accuracy and geometry don't always agree. A defense might look superior on an accuracy leaderboard but show massive representational drift when analyzed geometrically.
  • The $l_\infty$ Anomaly: In their experiments, networks trained against $l_\infty$ attacks showed the smallest accuracy gap (appearing very robust). However, geometric analysis revealed they had the largest mean displacement and the lowest neighborhood recovery. Essentially, the $l_\infty$ defense keeps the classification correct but allows the internal representation to drift significantly.
  • Training Trajectories as Artifacts: By using M-PHATE, the authors showed that adversarial training fundamentally alters the "path" a network takes during learning. Adversarially trained networks often see their representational stability collapse earlier in training compared to vanilla networks.
  • Defense-Specific Reshaping: Different adversarial objectives (FGSM, $l_2$, $l_\infty$) reshape the internal geometry in unique ways. This suggests that "robustness" isn't a single property, but a variety of structural changes to the latent space.

Key Results

The authors found that while adversarial training generally improves robustness, it always leads to structural changes in internal representations.

Figure 1: Clean accuracy and adversarial performance gaps for FGSM, ℓ2 and ℓ∞attacks across training perturbation budgets. A gap of 0 means clean accuracy matches accuracy under attack and larger gaps mean a larger drop under attack.

However, when looking at the geometric stability in the figure below, the story changes. The $l_\infty$ models, which looked best in the accuracy plots, show much lower "recovery" of their original neighborhoods compared to FGSM-trained models at higher perturbation budgets.

Figure 3: (a) Proportion of perturbed test points whose clean counterpart lies within its ten nearest neighbors in 3D PHATE space. (b) MMD permutation test p-values between clean and perturbed points in 3D PHATE space. Recovery near 1 and p near 1 both mean the perturbation left the representation essentially unchanged. For both panels, rows denote the trained network and applied evaluation attack, and columns denote the perturbation budget. Mean ± s.d. across three seeds.

Quantitatively, at "Extreme High" perturbation budgets, the $l_\infty$-trained network reduced the accuracy gap from 0.856 (vanilla) to 0.721, but its internal displacement was still significantly higher than the FGSM-trained counterpart. This proves that a defended network can maintain high accuracy even while its internal logic is being heavily distorted by the attack.

Limitations and Future Work

The primary limitation acknowledged by the authors is the scope: the study focused on small MLPs and the MNIST dataset using single-step attacks. While this is excellent for a foundational study, real-world applications involve much more complex architectures.

Future Work includes: * Scaling this geometric analysis to Convolutional Neural Networks (CNNs) and Transformers. * Testing on more complex datasets like CIFAR-10 and ImageNet, where representational structures are much richer. * Exploring whether these geometric shifts can be used as a real-time adversarial detection mechanism—if a point moves "too far" or "too weirdly" in latent space, the system could flag it as a potential attack.

Summary updated

Abstract

Adversarial training is a common technique for making neural networks more robust to small, intentionally crafted input perturbations, and robustness is commonly measured through accuracy under attack. By exploring inter-network representations across different adversarial training objectives, we can analyze whether multiple networks organize their latent representations similarly. We study Multilayer Perceptrons trained on MNIST under three adversarial training regimes and compare the resulting population training dynamics and geometry to those of a vanilla network. We use M-PHATE to embed data samples across training epochs and examine how the geometry of the data changes during training under adversarial objectives. On penultimate-layer representations we measure displacement between clean and perturbed points, distributional distance through maximum mean discrepancy, and nearest-neighbor recovery. Our visualizations and quantitative analyses show that adversarial training leads to structural changes in how networks organize internal data representations, and robust accuracy and representational stability can rank the same defenses in opposite ways. These findings support a discussion of re-evaluating how practitioners approach the auditing of model robustness by treating latent representations and their training trajectories as diagnostic artifacts across sets of networks.
BibTeX
@misc{ismalej2026representational,
  author = {Brandon Ismalej and Alex Cloninger and Gal Mishne},
  title = {{Representational Geometry as a Diagnostic for Adversarial Robustness}},
  year = {2026},
  eprint = {2611.17684},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17684},
}
Figure