When training models to predict or recognize movement, the most intuitive representation—absolute spatial coordinates—is often the most problematic. If a model learns that a "step forward" occurs at specific $(x, y, z)$ coordinates in a room, it may fail to recognize that same step when the person moves to a different corner or turns 90 degrees. This representational trap, where models accidentally memorize scene-specific shortcuts rather than general motion patterns, is what Christoph Schöller et al. call the "simplicity paradox" in What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction.
To build models that generalize across environments, researchers have converged on coordinate-invariant motion encoding. By shifting from a global "world" frame to local, relative, or body-fixed frames, these systems decouple the essence of a movement from its absolute location in space.
Breaking the Absolute Frame
The core intuition behind relative encoding is that the relationship between points is more informative than their absolute positions. For example, in robotic surgery, Hermann Mayer et al. found that representing tool trajectories through local coordinate frames allows a system to identify maneuvers like knot-tying regardless of where they occur in the workspace Skill Transfer and Learning by Demonstration in a Realistic Scenario of Laparoscopic Surgery.
This principle extends to human motion. Markos Diomataris et al. demonstrate that predicting pose changes relative to a local coordinate system makes motion models direction-agnostic, enabling them to reach goals from any starting orientation WANDR: Intention-guided Human Motion Generation. Similarly, Haopeng Lu et al. argue that aligning input and output coordinate systems through relative encoding prevents the network from needing to learn spatial invariance from scratch, significantly improving 3D pose estimation accuracy Improving Robustness and Accuracy via Relative Information Encoding in 3D Human Pose Estimation.
Root-Relative and Kinematic Representations
A common strategy for achieving this invariance is root-relative encoding, where all joint positions are calculated relative to a central anchor, such as the hip. Yinyu Nie et al. show that this disentanglement ensures learned motion patterns are invariant to an agent's global starting position, which is crucial for tasks like predicting room layouts from skeletal movement Pose2Room: Understanding 3D Scenes from Human Activities.
Beyond simple root-anchoring, the body can be represented as a series of local kinematic chains. Lorenzo Seidenari et al. propose expressing joint positions relative to their parent joints, which avoids mathematical singularities like gimbal lock while maintaining robustness to viewpoint changes Recognizing Actions from Depth Cameras as Weakly Aligned Multi-part Bag-of-Poses. Di Wu 0009 further refines this by using 3D positional pairwise differences to represent posture, ensuring the model remains person-centric and invariant to the subject's location in the sensor's field of view Leveraging Hierarchical Parametric Networks for Skeletal Joints Based Action Segmentation and Recognition.
Robustness to Dynamic Observers
Relative encoding is particularly powerful when the observer (the camera) is also moving. In multi-object tracking, absolute coordinates shift wildly during camera jerks or rotations. However, the relative spatial relationships between neighboring objects remain stable. Shuai Liu et al. leverage this "local geometry" to maintain object identities in drone-captured video, where erratic UAV motion would otherwise cause tracking failures Multi-Object Tracking Meets Moving UAV.
For video stabilization, Zhenmei Shi et al. represent camera motion as relative rotations rather than absolute coordinates. This allows the model to generalize across different environments by focusing on the patterns of shakiness (like a walking gait) rather than the specific direction the device is pointing Deep Online Fused Video Stabilization.
Geometric Invariants and Equivariance
The most extreme form of relative encoding involves geometric invariants—features that do not change under any rotation or translation. Ryoji Ogata et al. use Euclidean distances between joint pairs to classify exercise form, creating a model that is inherently viewpoint-invariant Temporal Distance Matrices for Squat Classification. Similarly, Fei Zhang et al. show that the changing distance between ankles or elbows provides a robust gait signature that works regardless of walking direction Exploring Relative Motion Features for Gait Recognition with Kinect.
Finally, some researchers hard-code these symmetries into the network architecture itself. Dian Wang et al. demonstrate that equivariant representations allow a robot to learn a task in one orientation and instantly generalize it to all others, eliminating the need for redundant training data across different spatial configurations Equivariant $Q$ Learning in Spatial Action Spaces. By treating motion as a set of relative displacements rather than absolute points, these methods allow models to focus on the "how" of movement rather than the "where."
Key Foundational Work
Skill Transfer and Learning by Demonstration in a Realistic Scenario of Laparoscopic Surgery (2003) — contributed 1 concept elements to this direction.
A Coordinate-Invariant Approach to Multiresolution Motion Analysis (2001) — contributed 1 concept elements to this direction.