TL;DR
This paper argues that standard accuracy metrics are insufficient for auditing adversarial robustness, as they fail to capture how a network's internal representation of data changes under attack. By using geometric analysis and dimensionality reduction (M-PHATE), the authors demonstrate that defenses which appear most robust by accuracy can actually be the most geometrically unstable, suggesting that representational geometry should be used as a primary diagnostic tool.
What Problem Does This Paper Address?
In the world of deep learning, "robustness" is the ability of a model to resist adversarial attacks—small, intentional perturbations designed to cause misclassification. Currently, the industry standard for measuring robustness is a single scalar: accuracy under attack.
However, the authors argue this is a "black box" metric. Two networks might achieve identical robust accuracy while organizing their internal data representations in completely different ways. Accuracy only tells us if a decision boundary was crossed; it doesn't tell us how much the internal "logic" of the network drifted or how the training process reshaped the data's geometry. Without understanding these structural changes, we lack a complete picture of why a defense works—or when it might fail.
Methods
The researchers moved beyond simple accuracy by treating neural network activations as geometric objects. They focused on the penultimate layer (the last layer before the final classification) of Multilayer Perceptrons (MLPs).
To visualize and quantify these high-dimensional changes, they employed several key techniques:
- M-PHATE: An extension of the PHATE dimensionality reduction method. It allows researchers to visualize "trajectories"—how the representation of data samples evolves epoch-by-epoch during training—and "joint embeddings" to see how clean and perturbed samples sit relative to each other in the same space.
- Geometric Metrics:
- Normalized Displacement ($d_{norm}$): Measures how far an adversarial perturbation pushes a data point in the latent space, relative to the typical distance between clean points.
- Maximum Mean Discrepancy (MMD): A statistical test to determine if the distribution of clean points and perturbed points are significantly different.
- k-Nearest Neighbor (kNN) Recovery: Measures if a perturbed point still stays close to its original "neighbors" in the latent space.
Datasets
The study utilized the MNIST dataset of handwritten digits. While a relatively simple dataset, it provided a controlled environment to establish a proof-of-concept for geometric diagnostics. The authors used a fixed batch of 1,000 training images and 2,000 test images across all experiments to ensure consistency.
Key Insights and Ideas
- The Accuracy-Geometry Paradox: The most striking finding is that accuracy and geometry don't always agree. A defense might look superior on an accuracy leaderboard but show massive representational drift when analyzed geometrically.
- The $l_\infty$ Anomaly: In their experiments, networks trained against $l_\infty$ attacks showed the smallest accuracy gap (appearing very robust). However, geometric analysis revealed they had the largest mean displacement and the lowest neighborhood recovery. Essentially, the $l_\infty$ defense keeps the classification correct but allows the internal representation to drift significantly.
- Training Trajectories as Artifacts: By using M-PHATE, the authors showed that adversarial training fundamentally alters the "path" a network takes during learning. Adversarially trained networks often see their representational stability collapse earlier in training compared to vanilla networks.
- Defense-Specific Reshaping: Different adversarial objectives (FGSM, $l_2$, $l_\infty$) reshape the internal geometry in unique ways. This suggests that "robustness" isn't a single property, but a variety of structural changes to the latent space.
Key Results
The authors found that while adversarial training generally improves robustness, it always leads to structural changes in internal representations.
However, when looking at the geometric stability in the figure below, the story changes. The $l_\infty$ models, which looked best in the accuracy plots, show much lower "recovery" of their original neighborhoods compared to FGSM-trained models at higher perturbation budgets.
Quantitatively, at "Extreme High" perturbation budgets, the $l_\infty$-trained network reduced the accuracy gap from 0.856 (vanilla) to 0.721, but its internal displacement was still significantly higher than the FGSM-trained counterpart. This proves that a defended network can maintain high accuracy even while its internal logic is being heavily distorted by the attack.
Limitations and Future Work
The primary limitation acknowledged by the authors is the scope: the study focused on small MLPs and the MNIST dataset using single-step attacks. While this is excellent for a foundational study, real-world applications involve much more complex architectures.
Future Work includes: * Scaling this geometric analysis to Convolutional Neural Networks (CNNs) and Transformers. * Testing on more complex datasets like CIFAR-10 and ImageNet, where representational structures are much richer. * Exploring whether these geometric shifts can be used as a real-time adversarial detection mechanism—if a point moves "too far" or "too weirdly" in latent space, the system could flag it as a potential attack.
Summary updated