When deep learning models process 3D data—such as point clouds, molecular structures, or meshes—they typically receive coordinates in a global Cartesian system $(x, y, z)$. This introduces a fundamental vulnerability: if an object is rotated or translated in space, its raw coordinate values change entirely.
To recognize a rotated object, a standard neural network must either undergo exhaustive data augmentation (training on every possible orientation) or use complex, computationally heavy equivariant architectures like spherical harmonics.
Local Reference Frame (LRF) alignment offers an elegant alternative. Instead of viewing the entire scene from a single, fixed global origin, the model constructs a unique, data-dependent coordinate system for each local neighborhood. Because these local frames are anchored directly to the geometry of the data, they rotate and translate with the object. By projecting raw coordinates into these local frames, the network achieves intrinsic rotation and translation invariance.
Constructing a Local Reference Frame
To build a local reference frame at a given point, we must define a stable, unique orthonormal basis—three mutually perpendicular unit vectors ($u, v, w$)—using only the local geometric context. If the object rotates globally, these vectors must rotate in the exact same manner.
Researchers have proposed several ways to extract these axes depending on the data structure:
- Statistical Neighborhoods (PCA): For unstructured point clouds, Park et al. demonstrate that applying Principal Component Analysis (PCA) to a point's local neighborhood yields three principal axes. These axes naturally adapt to the local spread of points, providing a robust local coordinate system.
- Differential Geometry: When dealing with continuous surfaces or manifolds, He et al. establish local frames by calculating the surface normal vector alongside the principal curvature axes of the manifold.
- Topological Anchors: For structured meshes, Liu et al. construct a canonical frame by leveraging the mesh topology itself, using a directed edge and its adjacent triangles to define the coordinate axes.
- Learned Orientations: Because purely geometric heuristics can be sensitive to noise, Luo et al. train a sub-network to predict equivariant vectors for each point based on its context, subsequently using the Gram-Schmidt process to orthogonalize them into a clean rotation matrix.
Global Frame (X, Y, Z) Local Reference Frame (u, v, w)
▲ ▲ w
│ │ /
│ │/──► u
└────────► ● local center
Orientation changes Rotates WITH the geometry;
raw coordinates. relative coordinates stay constant.
Achieving Invariance via Canonical Projection
Once a local reference frame is established for a neighborhood, it can be represented mathematically as a rotation matrix $R$. For a central point $p_{\text{center}}$ and a neighboring point $p_{\text{neighbor}}$, their raw global coordinates are transformed into the local frame via a simple projection:
$$p_{\text{local}} = R^T (p_{\text{neighbor}} - p_{\text{center}})$$
Because the matrix $R$ is bound to the local geometry, any global rotation applied to the entire object applies to $R$ as well. In the equation above, the global rotation cancels out, meaning $p_{\text{local}}$ remains completely identical regardless of how the object is oriented in the global space.
Rao et al. utilize this concept by mapping irregular 3D points onto a structured canonical frame before feeding them into a hierarchical network.
This projection can also be framed in relative spherical coordinates. For instance, Yan et al. and Li et al. transform Cartesian coordinates into relative radial distances, polar angles, and azimuthal angles within the local frame. This allows standard sequence models or quantum circuits to process 3D molecular geometries natively, bypassing the need for specialized equivariant network layers.
Aligning Multiple Frames for Message Passing
While LRFs grant perfect local invariance, they introduce a new challenge for deep learning architectures: frame misalignment. If every point in a point cloud or molecule calculates its own independent LRF, a feature vector computed at point $A$ cannot be directly compared or aggregated with a feature vector at point $B$, because they are defined in entirely different coordinate systems.
To allow information to flow across the graph via message passing, the network must align these disparate frames.
[ Local Frame A ] ───► Project Frame A onto Frame B ───► [ Local Frame B ]
Coordinates relative Coordinates relative
to Point A to Point B
To resolve this, Wang et al. introduce a mechanism that projects the orientation axes of one local coordinate system directly onto the axes of an adjacent local coordinate system. By tracking this relative rotation between neighboring frames, the network can pass geometric messages across multiple steps without losing track of spatial orientation. Similarly, Zhang et al. explicitly align local reference frame features before fusing them, ensuring that the aggregated global representation remains coherent and rotation-invariant.
Why the Idea Works
LRF alignment works because it successfully decouples pose from shape.
Instead of forcing a neural network to learn that a chair tilted at 45 degrees is the same as an upright chair, LRF alignment changes the camera angle for every local patch so that the network only ever sees the intrinsic geometry. By embedding the coordinate system into the data itself, the model becomes naturally immune to global translations and rotations.
As demonstrated across various domains—from face recognition (Lin et al.) and point cloud registration (Kadam et al.) to macromolecular modeling (Hu et al.)—this approach drastically reduces the hypothesis space the network needs to explore. The result is highly sample-efficient models that generalize perfectly to unseen 3D orientations out of the box.
Key Foundational Work
Automatic Correspondence for 3d Modeling: an Extensive Review (2005) — contributed 1 concept elements to this direction.
Fusion of Summation Invariants in 3D Human Face Recognition (2006) — contributed 1 concept elements to this direction.