Deep neural networks are often described as geometric engines. They take a tangled, messy manifold of input data and smoothly warp it layer by layer until the data becomes linearly separable or easily predictable. However, standard Euclidean distance fails to capture this process: two images might be pixel-wise almost identical (low Euclidean distance) but belong to entirely different classes, or they might look completely different but represent the same object.
To truly understand a network's internal logic, we need a coordinate-independent way to measure distance—one defined entirely by how the network itself perceives the data. This insight has driven a powerful research direction that applies Riemannian geometry to deep learning, using the network's own mathematical transformations to construct a localized map of its representation space.
The Core Mechanism: Pulling Back the Metric
Imagine placing a standard Euclidean ruler in a neural network's final output space. If we want to know how "far apart" two inputs are from the network's perspective, we shouldn't measure the distance between them in the raw input space. Instead, we should look at how much a tiny change in the input alters the final output.
By formalizing this intuition, we can "pull back" the metric from the output layer all the way to the input layer. Mathematically, if a network is represented as a smooth function $f(x)$, a small change in the input $dx$ produces a change in the output given by the Jacobian matrix $J = \frac{\partial f}{\partial x}$. The squared output distance is then:
$$\|df\|^2 = dx^T (J^T J) dx$$
The matrix $G(x) = J^T J$ acts as a local Riemannian metric tensor. As demonstrated by Hauser et al. (2017), this tensor can be systematically "backpropagated" through successive layers of a network.
The pullback metric provides a customized, localized ruler for every single point in the input space. As Mishne et al. (2023) show, calculating this local tensor allows researchers to precisely quantify how a network distorts distances and volumes across different regions of the input domain. If the network is highly sensitive to a certain feature, the metric tensor stretches the space in that direction; if it ignores a feature, the space is squished.
Equivalence Classes and Manifold Untangling
What happens when this custom ruler shrinks to zero in a specific direction? It means that moving the input along that path causes absolutely no change in the network's final predictions.
This phenomenon defines equivalence classes—continuous regions of the input space that the network treats as completely identical. Benfenati et al. (2022) and Benfenati et al. (2023) leverage singular Riemannian geometry to map these equivalence classes, providing a rigorous mathematical foundation for model interpretability. By understanding what a network collapses together, we can see exactly what features it has decided to ignore.
In later work, Benfenati et al. (2025) applied this framework to Transformers, developing algorithms to navigate input manifolds and discover alternative inputs that yield identical model predictions.
Input Space Output Space
[ Data Point A ] ---\
\---> [ Shared Representation / Class ]
[ Data Point B ] ---/
^
└─ Linked by an Equivalence Class (Zero Pullback Distance)
This collapsing of dimensions is fundamental to how networks learn invariances. For instance, an object recognition network needs to be invariant to the viewpoint of a camera. By tracking "view manifolds" across convolutional layers, Bakry et al. (2016) demonstrated that networks achieve viewpoint invariance by progressively collapsing these view manifolds in deeper layers while preserving task-relevant class boundaries.
Quantifying Expressivity Through Curvature
A network's capacity to represent complex functions—its expressivity—is intimately tied to the curvature of the manifolds it creates. If a network's representation space remains completely flat, it can only perform linear combinations of its inputs. To capture complex, non-linear real-world data, the network must introduce geometric curvature.
Using the pullback metric, researchers can compute the Riemann and sectional curvatures of a network's internal spaces. Chen et al. (2023) proved that the norm of the input-output Jacobian and the resulting manifold curvature serve as direct, quantifiable measures of an architecture's expressivity.
This geometric perspective enables powerful practical applications. For example, Asthana et al. (2025) introduced a zero-shot Neural Architecture Search (NAS) proxy that evaluates a network's potential performance without requiring any training or labeled data, simply by measuring the extrinsic curvature the untrained network induces on a simple geometric input manifold.
Geometry also explains why certain architectural innovations succeed where others fail. Standard attention mechanisms are inherently flat because they compute linear combinations of value vectors. However, Bathula et al. (2026) proved that adding multiplicative gating allows attention layers to generate intrinsic manifold curvature (measured via the Fisher-Rao metric), unlocking a fundamental boost in expressivity that scales quadratically with model depth.
Trajectories, Dynamics, and Network Comparison
Beyond static snapshots of trained models, Riemannian geometry provides a language for analyzing the dynamics of neural networks—both during a single forward pass and across the training timeline.
Instead of viewing a deep network as a black box, we can model its layer-by-layer computation as a trajectory moving along a high-dimensional manifold of neural representations. Lange et al. (2023) formalized this by applying Riemannian geometry to representational similarity metrics, showing that wider networks take more direct, efficient routes toward their target outputs.
This perspective extends naturally to recurrent architectures processing time-series data. Pellegrino et al. (2025) used a pullback metric to show that Recurrent Neural Networks (RNNs) compute by dynamically warping their internal state space over time, stretching relevant information to maintain sensitivity and squishing noise to ignore it.
Even the optimization landscape itself can be studied this way. Lucas et al. (2021) introduced the "Gauss length" to measure the curvature of a network's output trajectory as its weights are linearly interpolated, bridging weight-space optimization and function-space behavior.
Ultimately, these geometric properties dictate how well a model generalizes to new tasks. Cosentino et al. (2022) demonstrated that a self-supervised model's transfer learning success is heavily determined by the intrinsic dimensionality and curvature of the data manifolds it constructs. By shifting our focus from raw weight values to the intrinsic shape of activation manifolds, as advocated by Yu et al. (2018), researchers gain a coordinate-independent lens that reveals exactly how deep learning models structure, manipulate, and retain knowledge.
Key Foundational Work
Digging Deep into the Layers of CNNs: In Search of How CNNs Achieve View Invariance (2016) — contributed 2 concept elements to this direction.
Digging Deep into the layers of CNNs: In Search of How CNNs Achieve View Invariance Amr Bakry (2016) — contributed 1 concept elements to this direction.
Principles of Riemannian Geometry in Neural Networks (2017) — contributed 1 concept elements to this direction.