Continuous 3D Rotation Representations for Neural Networks

71 concepts58 papers

When training a deep neural network to predict spatial orientations—whether for a robotic arm, a drone, or a 3D human mesh—you will likely encounter a frustrating phenomenon: the network's loss suddenly spikes, or its predictions oscillate wildly during large rotations.

This instability is rarely a bug in the code. Instead, it stems from a fundamental mathematical mismatch between how neural networks learn and how traditional coordinate systems represent 3D rotations.

The Topological Mismatch

Deep neural networks are fundamentally continuous machines. They operate by mapping a continuous input space ($\mathbb{R}^n$) to a continuous output space via smooth matrix multiplications and activation functions. For a network to learn effectively, an infinitesimal change in the physical world (like a tiny tilt of a camera) must correspond to an infinitesimal change in the network’s output representation.

However, the space of 3D rotations—known mathematically as the special orthogonal group $SO(3)$—is a curved, compact manifold. It cannot be mapped to a flat Euclidean space of four or fewer dimensions without introducing topological discontinuities.

In a foundational paper, Zhou et al. proved that any 3D rotation representation with four or fewer dimensions is inherently discontinuous. These discontinuities act as "mathematical cuts" in the representation space. When a network must predict a smooth trajectory that crosses one of these cuts, it is forced to output a massive, instantaneous jump in numerical values for a tiny change in physical orientation. Because continuous neural networks struggle to approximate these abrupt step functions, training becomes unstable, gradients explode, and convergence slows.

Why Low-Dimensional Representations Fail

To build intuition for why this happens, let us look at the two most common low-dimensional rotation formats: Euler angles (3D) and quaternions (4D).

Euler Angles: Periodic Wrapping and Gimbal Lock

Euler angles represent rotations as three sequential turns around specified axes (e.g., roll, pitch, and yaw). This introduces two severe issues:

  1. Periodic Wrapping: A rotation of $0^\circ$ is physically identical to $360^\circ$. If an object rotates smoothly past the boundary, the target value must instantly jump from $360$ to $0$.
  2. Gimbal Lock: When two axes align, the system loses a degree of freedom, causing a mathematical singularity. As noted by Pavllo et al., these artifacts force networks to handle highly non-linear numerical jumps, leading to unstable gradients.

Quaternions: The Double-Cover Problem

Unit quaternions use four dimensions to avoid gimbal lock, but they introduce a different topological issue: antipodal ambiguity. Every physical 3D orientation corresponds to two distinct points on the 4D hypersphere: $q$ and $-q$.

This "double-cover" property confuses neural networks during training. If a network is uncertain and attempts to average its predictions between two equivalent orientations ($q$ and $-q$), the mathematical average lands at the origin $(0,0,0,0)$, which is an invalid, zero-norm quaternion. As Gilitschenski et al. and Schuck observe, this ambiguity creates conflicting gradient signals, particularly when navigating the full range of 3D rotations.

The Solution: Higher-Dimensional Continuity

To eliminate these topological cuts, we must embed the rotation manifold into a higher-dimensional space where it can "unfold" smoothly.

Zhou et al. proposed a simple, elegant 6D continuous representation. Instead of predicting angles or quaternions, the neural network outputs two unconstrained 3D vectors, $a_1$ and $a_2$. These six values are then mapped to a valid 3D rotation matrix $R = [b_1, b_2, b_3]$ using a deterministic, differentiable orthogonalization process (similar to Gram-Schmidt):

  1. Normalize the first vector to create the first axis: $$b_1 = \frac{a_1}{\|a_1\|}$$
  2. Subtract the projection of $a_2$ onto $b_1$ to ensure orthogonality, then normalize: $$b_2 = \frac{a_2 - (b_1 \cdot a_2)b_1}{\|a_2 - (b_1 \cdot a_2)b_1\|}$$
  3. Compute the cross product to get the final orthogonal axis: $$b_3 = b_1 \times b_2$$
Neural Network Output (6D) ──> [ a1 , a2 ]
                                 │    │
                                 ▼    ▼
Differentiable Orthogonalization (Gram-Schmidt / SVD)
                                 │
                                 ▼
Valid SO(3) Rotation Matrix ──> [ b1 , b2 , b3 ]

Because this mapping contains no cuts or boundaries, an infinitesimal change in the 6D output always results in an infinitesimal change in the final 3D rotation.

Alternatively, Esteves et al. demonstrated that a 9D representation can be used by letting the network predict all 9 elements of a $3 \times 3$ matrix, and then projecting it onto $SO(3)$ using Singular Value Decomposition (SVD). This approach provides an exceptionally stable optimization landscape because it treats the network's output as a continuous Euclidean space, completely bypassing the topological constraints of lower dimensions.

Alternative Paths: Tangent Spaces and Geometric Losses

While higher-dimensional embeddings solve the continuity problem by expanding the output space, other researchers tackle the issue by changing how the network interacts with the rotation manifold.

Lie Algebra (Tangent Spaces)

Instead of operating directly on the curved manifold $SO(3)$, a network can predict actions or states within the Lie algebra $\mathfrak{so}(3)$, which represents the flat tangent space at the identity rotation. As demonstrated by Schuck in reinforcement learning contexts, wrapping inputs and outputs in local tangent frames allows the network to explore and learn using standard Euclidean operations without encountering global topological cuts.

Manifold-Aware Loss Functions

If quaternions or matrices must be used, standard mean-squared error (MSE) losses will fail because they do not respect the geometry of the space. To mitigate this, researchers use specialized distributions. For instance, Gilitschenski et al. utilized the Bingham distribution to design a loss function that inherently recognizes antipodal symmetry ($q \equiv -q$), ensuring that the network receives consistent gradient signals regardless of which quaternion sign is chosen during training.

Conclusion

The core takeaway is intuitive: neural networks require continuous target spaces to learn smoothly. When forcing a network to predict 3D orientations, abandoning low-dimensional representations (Euler angles, quaternions) in favor of continuous 6D or 9D representations removes the artificial mathematical boundaries that disrupt backpropagation. By ensuring that small physical changes always equal small numerical changes, models converge faster, achieve higher precision, and remain stable across the entire sphere of rotation.

Key Foundational Work

Predicting Slice-to-Volume Transformation in Presence of Arbitrary Subject Motion (2017) — contributed 1 concept elements to this direction.
3D Reconstruction in Canonical Co-ordinate Space from Arbitrarily Oriented 2D Images (2017) — contributed 1 concept elements to this direction.
Control of a Quadrotor with Reinforcement Learning (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_rotation_representation_discontinuity_7bd2a3b7,
  title = {rotation representation discontinuity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rotation-representation-discontinuity/rdr_7bd2a3b77cc342b0b8bd70623b13a526},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_rotation_representation_discontinuity_7bd2a3b7_v1,
  title = {rotation representation discontinuity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rotation-representation-discontinuity/rdr_7bd2a3b77cc342b0b8bd70623b13a526/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake