Batch Normalization on Riemannian Manifolds

This direction is merged into another direction: geodesic computation overhead. Its last version stays readable and citable here.

Version 1 of 135 papers55 conceptsfrozen 2026-10-04

In standard deep learning, Batch Normalization (BN) is the engine of stability. It ensures that activations don't explode or vanish by centering them around a zero mean and scaling them to unit variance. However, this process relies entirely on Euclidean arithmetic: adding vectors to find an average and subtracting that average to center the data.

When data lives on a Riemannian manifold—such as the hyperbolic space used for hierarchies, the sphere for directional data, or Symmetric Positive Definite (SPD) matrices for neuroimaging—Euclidean arithmetic fails. If you take the arithmetic mean of two points on a sphere, the resulting point lies inside the sphere, not on its surface. To build deep networks that operate natively on these curved surfaces, we must reinvent normalization to respect the underlying geometry.

The Geometry of the Mean

The first step in normalization is finding the "center" of a batch. In Euclidean space, this is the arithmetic mean. On a manifold, we replace this with the Fréchet mean (also known as the Riemannian barycenter).

As Rudrasis Chakraborty explains in ManifoldNorm: Extending normalizations on Riemannian Manifolds, the Fréchet mean is the point on the manifold that minimizes the sum of squared geodesic distances to all samples in a batch. Unlike the arithmetic mean, the Fréchet mean is guaranteed to stay on the manifold.

Early implementations, such as those by Aaron Lou and Isay Katsman in Differentiating through the Fréchet Mean, provided the mathematical framework to backpropagate through this optimization process. This allowed hyperbolic graph networks and manifold-valued models to maintain geometric consistency during training.

Centering via Parallel Transport

Once we have the mean, how do we "subtract" it? On a curved surface, you cannot simply subtract one point from another. Instead, we use Parallel Transport.

Think of parallel transport as sliding a vector along the manifold's surface without changing its direction relative to the surface. In Riemannian batch normalization for SPD neural networks, Daniel A. Brooks and colleagues demonstrate how to use this to center data. They transport each data point from the batch mean to a fixed reference point (like the identity matrix for SPD manifolds). This geometric shift acts as the non-Euclidean analog to subtraction, standardizing the location of the data distribution without distorting its internal structure.

Scaling in the Tangent Space

After centering, we must scale the variance. Because the manifold itself is curved, variance is typically measured and adjusted in the tangent space—a flat, Euclidean-like plane that touches the manifold at a specific point.

Ziheng Chen and Nicu Sebe have developed unified frameworks like LieBN and GyroBN to handle this. By mapping data into the tangent space of the identity element, they can perform standard scaling (division by the standard deviation) and then map the data back onto the manifold. This ensures that the "spread" of the features is controlled, preventing the numerical instabilities that often plague deep non-Euclidean networks.

From Iterative Solvers to Closed-Form Speed

A major hurdle for Riemannian BN was computational cost. Calculating the Fréchet mean usually requires an iterative optimization loop, which is slow. Recent breakthroughs have focused on finding "closed-form" approximations that provide the benefits of normalization at a fraction of the cost.

In Poincaré ResNet, Pascal Mettes suggests replacing the iterative mean with a simple midpoint calculation. Similarly, Ahmad Bdeir and Niels Landwehr utilize the Lorentzian centroid in Fully Hyperbolic Convolutional Neural Networks for Computer Vision. These closed-form statistics allow models like Lorentzian Residual Neural Networks to achieve massive speedups—up to 2,000x in some cases—making deep hyperbolic learning practical for large-scale tasks.

Why It Works: Solving the Boundary Problem

Why go through all this mathematical trouble? The primary reason is the boundary problem. In hyperbolic models like the Poincaré disk, as data points move toward the edge, the distance between them grows exponentially. Without normalization, gradients vanish and numerical precision collapses as points drift toward this "infinity" boundary.

By using manifold-aware normalization, researchers like Xianglong Shi in Intrinsic Lorentz Neural Network keep activations centered in the "sweet spot" of the manifold. This stability allows for the training of much deeper architectures, enabling hyperbolic models to outperform massive Euclidean transformers in domains with intrinsic hierarchies, such as genomics and neuroimaging.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_batch_normalization_on_riemannian_manifo_5abaf0d9,
  title = {Batch Normalization on Riemannian Manifolds},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/batch-normalization-on-riemannian-manifolds/rdr_5abaf0d92b604a2794d1d26286aed13a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_batch_normalization_on_riemannian_manifo_5abaf0d9_v1,
  title = {Batch Normalization on Riemannian Manifolds},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/batch-normalization-on-riemannian-manifolds/rdr_5abaf0d92b604a2794d1d26286aed13a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}