Geodesic Distance Losses for 3D Orientation Learning

24 concepts17 papers1 proposal

Standard deep learning frameworks rely heavily on the assumption of flat, Euclidean vector spaces. When computing a loss function, we routinely subtract a prediction from a target and calculate the mean squared error. However, many real-world physical properties—most notably 3D rotations and orientations—do not live in flat space. Instead, they reside on curved, compact Riemannian manifolds such as the Special Orthogonal Group $SO(3)$ or the Special Euclidean Group $SE(3)$.

Treating these curved spaces as flat vectors introduces severe geometric distortions that degrade model performance. To resolve this, a prominent research direction focuses on replacing Euclidean proxy metrics with geodesic distance losses that natively respect the non-Euclidean geometry of rotation spaces.

The Failure of Euclidean Metrics in Curved Spaces

To understand why standard loss functions fail, consider representing a 3D rotation as a quaternion (a point on a hypersphere) or a $3 \times 3$ matrix. If a neural network outputs a prediction in this format, a standard Euclidean loss measures the straight-line distance (a chord) cutting directly through the interior of the geometry, rather than traveling along its surface.

As Adam M. Smith et al. formalized in Learning Rotations, traditional error metrics fail to account for the specific spherical geometry inherent in rotation problems, leaving a significant gap in how we guide updates within the algebraic constraints of Lie groups. When models ignore this geometry, they suffer from optimization artifacts. For instance, Yannan He et al. point out in NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors that treating rotations as Euclidean vectors fails to account for the periodic, wrapping nature of $SO(3)$, which causes slow convergence and introduces optimization artifacts.

Furthermore, this geometric mismatch distorts uncertainty estimation. As Igor Gilitschenski et al. demonstrate in Deep Orientation Uncertainty Learning based on a Bingham Loss, applying standard Gaussian distributions or Euclidean metrics to orientation tasks yields mathematically incorrect distance measurements and poor uncertainty bounds, particularly in scenarios where the data is highly dispersed.

The Geodesic Alternative

A geodesic is the generalization of a straight line to curved spaces—it represents the shortest path between two points along the surface of a manifold. In the context of $SO(3)$, the geodesic distance corresponds exactly to the true angular displacement between two orientations.

Euclidean Loss (Chord)        Geodesic Loss (Arc)
      _______                       _______
     /   |   \                     /   ^   \
    /    |    \                   /    |    \
   P*----+----* T                P*=========>* T
    \         /                   \         /
     \_______/                     \_______/
  (Cuts through manifold)       (Follows manifold surface)

Optimizing for this true angular distance fundamentally changes the gradient landscape. Siddharth Mahendran et al. leveraged this property in 3D Pose Regression Using Convolutional Neural Networks to shift 3D object pose estimation from discrete classification to continuous regression. By utilizing geodesic loss functions, the authors demonstrated that deep networks can effectively navigate non-Euclidean geometry to predict precise orientations.

The core reason this approach succeeds is alignment. As the authors later elaborated in A Mixed Classification-Regression Framework for 3D Pose Estimation from 2D Images, geodesic loss ensures that the training objective aligns perfectly with the actual geometric properties of the rotation group, preventing the suboptimal convergence typical of squared error losses.

This alignment becomes especially critical when dealing with large errors. In Real-Time Deep Pose Estimation With Geodesic Loss for Image-to-Template Rigid Registration, Seyed Sadegh Mohseni Salehi et al. showed that employing a geodesic loss function significantly enhances accuracy for larger rotational differences. While Euclidean metrics saturate or provide distorted gradients when predictions are far from the target, the geodesic metric maintains a physically meaningful error signal. This property is vital for highly constrained physical systems, such as human joint tracking. As Liangyan Gui et al. showed in Adversarial Geometry-Aware Human Motion Prediction, modeling joint rotations via geodesic distance on the $SO(3)$ manifold prevents the physical and mathematical inconsistencies that occur when treating joint angles as simple vectors in flat space.

Coordinate Invariance and Tangent Space Backpropagation

Geodesic losses also introduce crucial algebraic symmetries, such as coordinate invariance. When tracking a robot or a camera in 3D space, the calculated error should not change simply because the global coordinate frame was rotated or translated.

In NeRF-VIO: Map-Based Visual-Inertial Odometry with Initialization Leveraging Neural Radiance Fields, Yanyu Zhang et al. utilize a left-invariant loss function based on geodesic distance on the $SE(3)$ manifold. This geometry-aware formulation ensures consistent error calculation across different coordinate frames, allowing the model to handle the non-linear coupling of rotation and translation seamlessly.

However, backpropagating through a geodesic loss presents a technical challenge: standard automatic differentiation frameworks operate on flat vector spaces. If you update a rotation matrix by taking a standard gradient step, the resulting matrix will no longer be orthogonal, breaking the fundamental constraints of $SO(3)$.

To bypass this, Zachary Teed et al. developed LieTorch in Tangent Space Backpropagation for 3D Transformation Groups. Instead of performing optimization directly on the curved manifold, the network projects errors into the tangent space—a local vector space (Lie algebra) centered at the current estimate. Because the tangent space is locally flat, standard calculus and backpropagation work perfectly. The updated vector is then mapped back onto the curved manifold using the exponential map. This framework provides numerically stable, representation-agnostic gradients, completely avoiding numerical explosions and singularities.

Extending to Probabilistic and Generative Models

The intuition behind geodesic optimization has expanded beyond simple regression into representation learning, uncertainty estimation, and generative modeling.

Manifold Embeddings and Uncertainty

By embedding geodesic distances directly into feature matching, networks can learn better structured latent spaces. Mai Bui et al. demonstrated in When Regression Meets Manifold Learning for Object Recognition and Pose Estimation that scaling triplet loss penalties based on the true angular distance allows a network to map the continuous 3D manifold of rotations into a highly coherent feature space.

For probabilistic tracking, Fahira Afzal Maken et al. introduced specialized radial basis function kernels designed for angular distances in Stein ICP for Uncertainty Estimation in Point Cloud Matching, ensuring that the attractive and repulsive forces in variational frameworks accurately reflect the true geometry of 3D transformations. Similarly, Akash Sengupta et al. designed normalizing flows directly on $SO(3)$ in HuManiFlow: Ancestor-Conditioned Normalising Flows on SO(3) Manifolds for Human Pose and Shape Distribution Estimation, allowing models to output complex, multi-modal probability distributions over orientations without vector-space artifacts.

Generative Flow and Diffusion

In generative modeling, standard noise injection and velocity prediction fail on curved surfaces because straight lines drift off the manifold. To build diffusion models on $SO(3)$, Yesukhei Jagvaral et al. adapted score-matching losses in Unified Framework for Diffusion Generative Models in SO(3): Applications in Computer Vision and Astrophysics by utilizing the Isotropic Gaussian on $SO(3)$ ($IGSO(3)$) as a native noise kernel, enabling score-based models to operate effectively in rotational spaces.

For flow matching, Olga Zaghen et al. proved in Riemannian Variational Flow Matching for Material and Protein Design that predicting local velocity vectors on a curved manifold is mathematically inferior to predicting endpoints. By reformulating the objective to minimize the squared geodesic distance between the predicted and target endpoints, the authors provided a robust learning signal that allows generative models to navigate complex, non-Euclidean structures smoothly.

Key Foundational Work

3D Pose Regression Using Convolutional Neural Networks (2017) — contributed 2 concept elements to this direction.
Learning Rotations (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geodesic_manifold_loss_53d2b1b5,
  title = {geodesic manifold loss},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geodesic-manifold-loss/rdr_53d2b1b5866a474db0b8ae8590387f00},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_geodesic_manifold_loss_53d2b1b5_v1,
  title = {geodesic manifold loss},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geodesic-manifold-loss/rdr_53d2b1b5866a474db0b8ae8590387f00/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake