The geometry of neural network decision boundaries

Dylan M Paiton, David L Schultheiss, Matthias Kümmerer, Zac Cranko, Matthias BethgearXiv 2026 · arXiv

TL;DR

This paper introduces a principled method to measure and visualize neural network decision boundaries directly in high-dimensional space, avoiding the distortions of traditional 2D projections. The authors find that decision boundaries are surprisingly close to natural inputs across many dimensions and that adversarial training significantly "flattens" and pushes these boundaries away, a change driven more by training methodology than by model architecture.

What Problem Does This Paper Address?

Deep neural networks are often criticized as "black boxes." While we understand their mathematical operations, intuitively grasping why they make specific decisions remains difficult. Central to this understanding is the decision boundary: the surface in input space where a model's classification changes.

Visualizing these boundaries is notoriously hard because they exist in high-dimensional spaces (e.g., 3,072 dimensions for a CIFAR-10 image). Previous attempts often relied on projecting these surfaces onto 2D planes, which distorts distances and can map different high-dimensional points to the same 2D location. Furthermore, there has been conflicting theory about the "shape" of these boundaries—whether adversarial examples are isolated "pockets" or if the boundary is a smooth, continuous manifold.

Methods

The authors propose a two-pronged approach to study boundary geometry:

  1. Defining Subspaces: Instead of random projections, they find a set of orthogonal directions ($\delta_1, \delta_2, \dots, \delta_m$) that point to the closest decision boundaries. They use an untargeted adversarial attack framework, optimized via an interior-point method, to find these "minimal distance" directions.
  2. Measuring Curvature: They treat the decision boundary as a Riemannian manifold. Using tools from differential geometry, they derive a formula for the "shape operator" to calculate principal curvatures. This allows them to quantify whether a boundary is flat, bowing inward (positive curvature), or bowing outward (negative curvature). Crucially, they show that previous attempts in the literature to measure this curvature were mathematically incorrect for high-dimensional level-sets.

They also introduce a 2D visualization technique that takes a "slice" of the high-dimensional space spanned by two adversarial directions. Unlike a projection, this preserves exact spatial relationships within that slice.

Figure 1: The decision boundary for an adversarially trained ResNet-50. (Left) A subspace centered on a dataset image (class 5 exemplar, bottom middle) is defined, with each orthogonal axis corresponding to a direction where the decision boundary is closest. Principal curvature is measured at random points within the subspace. (Right) A 2-dimensional subspace is visualized using two adversarial directions, accurately showing the boundary shape and distances to alternate class labels. Perturbation images outside the graph are rescaled for visualization, with their ℓ2 length indicated on the axes. Each color represents a class (e.g., deer: class 4, dogs: class 5, horses: class 7). White arrows indicate sampling directions within the subspace to locate the boundary.

Datasets

The study evaluates models trained on:

  • CIFAR-10: Using ResNet-50 and WideResNet-70 architectures.
  • MNIST: Used primarily for consistency checks and validating the mathematical derivations.

They compare "natural" models (standard training) against "robust" models (adversarial training).

Key Insights and Ideas

  • Dense Adversarial Regions: The research supports the hypothesis that adversarial examples occupy dense regions of the input space. Boundary distances grow smoothly and slowly as the dimensionality of the search space increases, refuting the idea that adversarial examples are just "isolated spikes."
  • Training Trumps Architecture: Interestingly, the geometry of the decision boundary is almost entirely determined by the training method (natural vs. adversarial). A ResNet-50 and a WideResNet-70 trained the same way look remarkably similar geometrically, despite their different capacities.
  • Linearization through Robustness: Adversarial training doesn't just push the boundary further away; it linearizes it. Robust models exhibit much lower curvature (flatter boundaries) near natural images compared to naturally trained models.
  • Label Diversity: In robust models, the local neighborhood of an image contains a much more diverse set of "alternate" class labels. In natural models, the closest boundary usually leads to only one or two specific "error" classes.

Key Results

  • Boundary Distance: In naturally trained CIFAR-10 models, the average $\ell_2$ distance to the boundary is very small (~0.1). Adversarial training increases this distance by an order of magnitude (~1.5 to 2.0).
  • Sub-linear Growth: As you look for boundaries in more orthogonal directions, the distance increases sub-linearly. This means there are many "weak spots" near any given image.
Figure 2: Boundary distances. The distance from a test set image to the decision boundary grows sub-linearly with increasing subspace dimensionality and remains consistent across model architectures. (Left, top) Box plots show decision boundary distances for uniformly sampled directions within n-dimensional subspaces (horizontal axis), with dots indicating distances along the subspace axes, averaged across images. Boxes show interquartile ranges, whiskers represent the 10% and 90% percentiles, and dashed lines show random sample means. Every other dimension up to 50 is shown. (Left, bottom) Similar to the top panel, but the vertical axis shows the largest boundary distance within the subspace divided by the largest boundary distance along the subspace axes. Low values indicate that the boundary remains close to the input sample between the axes. (Right) Histograms of boundary distances for randomly sampled directions within subspaces, with dimensionality increasing across subplots (odd-numbered dimensions are shown). The mean shifts rightward and variance increases proportionately with dimensionality, suggesting a smooth growth in distance with increasing dimensionality. Line colors represent training schemes as per the legend.
  • Curvature Profiles: Naturally trained models show significant negative mean curvature (bowing outward), while robust models are nearly flat (mean curvature near zero).
Curvature profiles showing that robust models (red) have much flatter boundaries than natural models (blue).

Limitations and Future Work

  • Computational Cost: Finding 50+ orthogonal adversarial directions per image is computationally expensive, limiting the scale of the analysis to smaller datasets like CIFAR-10.
  • Activation Functions: The curvature math requires twice-differentiable functions. The authors used ELU activations; while they found this didn't change accuracy, the behavior of non-smooth activations like ReLU remains an area for future investigation.
  • Perceptual Gap: While the paper measures geometric distance, it notes that "perceptual distance" (how a human perceives the change) is still an open research question. Some large geometric perturbations in robust models still look like the original class to humans.

Summary updated

Abstract

Artificial neural networks (ANNs) are built using well-understood mathematical operations, yet their complex nature poses significant challenges to intuitively understanding their decision-making processes. This work introduces a method for visualizing and measuring ANN decision boundary geometry directly in the high-dimensional input space, avoiding distortions found with traditional projection-based techniques. We analyze boundary curvatures and distances in diverse inputs, architectures, and training methods. Boundary distances grow smoothly and slowly with consistent curvature as the size of the subspace increases, suggesting that decision boundaries are close to input samples within large, continuous subspaces. We also quantify how much boundary distances and curvatures reduce near dataset samples after adversarial training. Training methodology has a significantly stronger impact on boundary distance and curvature than architecture or test-set accuracy. These findings are contextualized within prior work, supporting or refuting several previously proposed theories of boundary behavior. Our analytically and empirically supported methodology lays the foundation for future studies to improve network interpretability and understanding.
BibTeX
@misc{paiton2026geometry,
  author = {Dylan M Paiton and David L Schultheiss and Matthias Kümmerer and Zac Cranko and Matthias Bethge},
  title = {{The geometry of neural network decision boundaries}},
  year = {2026},
  eprint = {2611.24557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24557},
}
Figure