TL;DR
This paper introduces a principled method to measure and visualize neural network decision boundaries directly in high-dimensional space, avoiding the distortions of traditional 2D projections. The authors find that decision boundaries are surprisingly close to natural inputs across many dimensions and that adversarial training significantly "flattens" and pushes these boundaries away, a change driven more by training methodology than by model architecture.
What Problem Does This Paper Address?
Deep neural networks are often criticized as "black boxes." While we understand their mathematical operations, intuitively grasping why they make specific decisions remains difficult. Central to this understanding is the decision boundary: the surface in input space where a model's classification changes.
Visualizing these boundaries is notoriously hard because they exist in high-dimensional spaces (e.g., 3,072 dimensions for a CIFAR-10 image). Previous attempts often relied on projecting these surfaces onto 2D planes, which distorts distances and can map different high-dimensional points to the same 2D location. Furthermore, there has been conflicting theory about the "shape" of these boundaries—whether adversarial examples are isolated "pockets" or if the boundary is a smooth, continuous manifold.
Methods
The authors propose a two-pronged approach to study boundary geometry:
- Defining Subspaces: Instead of random projections, they find a set of orthogonal directions ($\delta_1, \delta_2, \dots, \delta_m$) that point to the closest decision boundaries. They use an untargeted adversarial attack framework, optimized via an interior-point method, to find these "minimal distance" directions.
- Measuring Curvature: They treat the decision boundary as a Riemannian manifold. Using tools from differential geometry, they derive a formula for the "shape operator" to calculate principal curvatures. This allows them to quantify whether a boundary is flat, bowing inward (positive curvature), or bowing outward (negative curvature). Crucially, they show that previous attempts in the literature to measure this curvature were mathematically incorrect for high-dimensional level-sets.
They also introduce a 2D visualization technique that takes a "slice" of the high-dimensional space spanned by two adversarial directions. Unlike a projection, this preserves exact spatial relationships within that slice.
Datasets
The study evaluates models trained on:
- CIFAR-10: Using ResNet-50 and WideResNet-70 architectures.
- MNIST: Used primarily for consistency checks and validating the mathematical derivations.
They compare "natural" models (standard training) against "robust" models (adversarial training).
Key Insights and Ideas
- Dense Adversarial Regions: The research supports the hypothesis that adversarial examples occupy dense regions of the input space. Boundary distances grow smoothly and slowly as the dimensionality of the search space increases, refuting the idea that adversarial examples are just "isolated spikes."
- Training Trumps Architecture: Interestingly, the geometry of the decision boundary is almost entirely determined by the training method (natural vs. adversarial). A ResNet-50 and a WideResNet-70 trained the same way look remarkably similar geometrically, despite their different capacities.
- Linearization through Robustness: Adversarial training doesn't just push the boundary further away; it linearizes it. Robust models exhibit much lower curvature (flatter boundaries) near natural images compared to naturally trained models.
- Label Diversity: In robust models, the local neighborhood of an image contains a much more diverse set of "alternate" class labels. In natural models, the closest boundary usually leads to only one or two specific "error" classes.
Key Results
- Boundary Distance: In naturally trained CIFAR-10 models, the average $\ell_2$ distance to the boundary is very small (~0.1). Adversarial training increases this distance by an order of magnitude (~1.5 to 2.0).
- Sub-linear Growth: As you look for boundaries in more orthogonal directions, the distance increases sub-linearly. This means there are many "weak spots" near any given image.
- Curvature Profiles: Naturally trained models show significant negative mean curvature (bowing outward), while robust models are nearly flat (mean curvature near zero).
Limitations and Future Work
- Computational Cost: Finding 50+ orthogonal adversarial directions per image is computationally expensive, limiting the scale of the analysis to smaller datasets like CIFAR-10.
- Activation Functions: The curvature math requires twice-differentiable functions. The authors used ELU activations; while they found this didn't change accuracy, the behavior of non-smooth activations like ReLU remains an area for future investigation.
- Perceptual Gap: While the paper measures geometric distance, it notes that "perceptual distance" (how a human perceives the change) is still an open research question. Some large geometric perturbations in robust models still look like the original class to humans.
Summary updated