When a human reaches for a coffee cup, they don't calculate the cup's position in "GPS coordinates" relative to the center of the Earth. Instead, they perceive the cup's position relative to their hand. This distinction—absolute versus relative representation—is the difference between a robot that can only perform a task in a specific lab setting and one that can generalize to any room in the world.
A growing body of research suggests that the key to robust, zero-shot generalization in embodied AI is the systematic replacement of absolute global coordinates with relative, robot-centric, or object-centric frames of reference.
The Problem with Absolute Coordinates
In many early reinforcement learning (RL) setups, the state space is defined by absolute world coordinates ($x, y, z$). While mathematically simple, this creates a significant hurdle for generalization. As noted in Observation Space Matters: Benchmark and Optimization Algorithm, including absolute coordinates that increase indefinitely can cause a policy to fail when it encounters "out-of-distribution" values—essentially, the agent finds itself at a coordinate it never saw during training and doesn't know how to react.
When a model learns in absolute terms, it "memorizes" the geometry of a specific workspace. If the robot or the target object is moved by just a few meters, the input features change entirely, even if the underlying task logic remains identical.
Shifting the Frame of Reference
To overcome this, researchers like Mikhail Pavlov and colleagues demonstrated in Run, skeleton, run: skeletal model in a physics-based simulation that transforming spatial coordinates into relative ones centered on the agent allows learned behaviors to generalize across different environment scales.
This shift provides several immediate benefits:
- Translation Invariance: If the entire scene moves, the relative distances remain the same, and the policy continues to work.
- Dimensionality Reduction: As Aloke Guha argued in early work on control scheduling, representing states relative to a moving target reduces the complexity of the state space the agent must explore.
- Geometric Consistency: Kevin Huang showed in DATT that transforming inputs into a robot-centric body frame is essential for stable training and tracking arbitrary trajectories.
Local Frames for Manipulation
In robotic manipulation, the "local frame" often centers on the robot's end-effector or the object itself. Stelian Coros and Marco Bagatella found that a hand-centric coordinate system effectively increases training data density. By treating every interaction as a relative movement between the hand and the object, the model learns that a "grasp" looks the same whether it happens on the left or right side of a table (Problem Space Transformations for Generalisation in Behavioural Cloning).
Similarly, Yajvan Ravan and the team at MIT developed the PoPi framework, which represents states in a local frame relative to both the robot and the manipulated object. This abstraction allowed their policy to achieve zero-shot generalization, moving office chairs through novel room layouts it had never encountered during training.
Delta Actions and Residual Logic
The principle of relativity applies to actions as well as observations. In Demystifying Action Space Design for Robotic Manipulation Policies, Yuchun Feng and co-authors conducted a massive study of 13,000+ rollouts, concluding that delta (relative) actions are fundamentally superior to absolute ones.
By predicting the change in position ($\Delta x$) rather than the target position ($x_{target}$), the neural network's mapping complexity is reduced. This "residual representation" acts as a powerful inductive bias, as seen in DynScene by Sangmin Lee, where decoupling task logic from global coordinates allowed for the generation of diverse, physically plausible robotic trajectories.
Beyond Robotics: Scaling and Logic
The intuition of relative representation extends to combinatorial optimization and symbolic reasoning. Thomas D Barrett showed that representing search progress relative to the "best-so-far" solution allows RL agents to solve problems 50x larger than those they were trained on (ECO-DQN).
In the realm of symbolic AI, the QORA algorithm achieves zero-shot transfer by learning transition rules based on relative object relationships. Because the rules are defined by how objects relate to one another (e.g., "A is to the left of B") rather than where they sit in a grid, the logic remains valid regardless of the environment's total size.
Conclusion
By anchoring representations to the agent, the tool, or the target, researchers are moving away from "memorized" spatial control toward "principled" geometric interaction. Whether it is a quadrotor tracking a trajectory in a body frame or a robotic arm performing a precise insertion via relative 6D poses, the shift toward relative representations provides the invariance necessary for AI to step out of the simulator and into the unpredictable, wide-scale real world.