Traditional imitation learning and reinforcement learning often define robotic policies directly within the robot's joint space or relative to a fixed robot-base frame. While straightforward, this approach tightly couples the learned behavior to a specific hardware configuration. If you change the link lengths, add a joint, or attempt to learn from a human video demonstration, the policy fails due to the embodiment gap.
To bypass this limitation, a powerful paradigm has emerged: shifting the focus from how the robot moves to how the environment changes. By framing states and actions in relative, coordinate-free task spaces, policies become inherently embodiment-agnostic.
The Power of Relative Invariance
Why does this shift work? Geometry is invariant, whereas kinematics are platform-specific. If a task is defined as moving an end-effector relative to an object rather than moving joints relative to a base, the underlying mechanical structure of the robot becomes irrelevant.
For instance, Wilcox et al. show in Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learning that transforming visual observations directly into an end-effector-centric coordinate frame makes the policy invariant to the robot arm's specific joint configurations. This mathematical transformation enables zero-shot transfer across entirely different physical platforms. Similarly, Ramesh et al. demonstrate in Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies that formulating policies in the $SE(3)$ pose space rather than joint coordinates abstracts the task logic away from hardware kinematics, facilitating seamless cross-platform deployment.
Object-Centric Trajectories and Task Graphs
When the robot's physical structure is abstracted away, object trajectories and spatial relationships serve as a universal intermediate language. If a policy specifies how an object should move, any robot can execute that trajectory using standard inverse kinematics (IK).
Argus proposes in DITTO: Demonstration Imitation by Trajectory Transformation that manipulation tasks can be represented purely as a sequence of 3D object poses relative to the environment. By focusing entirely on the object's path rather than human skeletal movements or robot joint angles, the system completely sidesteps the human-to-robot correspondence gap. To scale this across complex scenes, Zeng et al. develop GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes, which synthesizes 6-DOF object trajectories that can be mapped onto any robotic embodiment.
[Human Video] ──> [Extract Object-Centric Graph/Trajectory] ──> [Inverse Kinematics] ──> [Any Robot Joint Space]
When tasks involve intricate geometries or multiple objects, a single global object pose is insufficient. Instead, the scene can be modeled as a structured graph of relative spatial constraints:
- Visual Entity Graphs: Sieb et al. introduce this concept in Graph-Structured Visual Imitation, combining coarse bounding boxes for location with fine-grained point features for orientation. This allows a robot to transfer skills to objects of different geometries by simply updating the entity detection nodes.
- Open-World Object Graphs: For unconstrained environments, Zhu et al. leverage these graphs in Vision-based Manipulation from Single Human Video with Open-World Object Graphs to capture object-to-object relationships from a single human video, completely ignoring the specific morphology of the demonstrator.
- Affordance Regions: Plonka et al. show in Learning Spatial Bimanual Action Models Based on Affordance Regions and Human Demonstrations that defining tasks through the spatial relationships between functional object parts ensures the manipulation strategy remains decoupled from the agent's physical structure.
Scaling to Bimanual Coordination and Dexterity
The benefits of relative task-space representations become even more pronounced in bimanual manipulation and dexterous control, where coordinating high-dimensional joint spaces is notoriously difficult. By parameterizing actions as the relative poses between hands and objects, complex coordination patterns become highly portable.
In Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks, Gao et al. extract embodiment-independent hand-object relationships to ensure skills remain robust to workspace clutter. Similarly, Zhou et al. demonstrate in One-Shot Real-World Demonstration Synthesis for Scalable Bimanual Manipulation and BiNoMaP: Learning Category-Level Bimanual Non-Prehensile Manipulation Primitives that training policies using 6-DoF end-effector poses relative to object frames—rather than joint coordinates—enables zero-shot transfer across entirely different dual-arm hardware configurations. To anchor these skills to novel objects without retraining, Zhou et al. also propose VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Bimanual Robotic Manipulation, using vision-language models to ground relative spatial relationships across different robot bodies.
This abstraction extends down to fine-grained dexterous manipulation. Tao et al. show in DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies that tracking fingertip trajectories relative to a palm-centric frame allows dexterous manipulation policies to transfer across distinct robotic hand architectures. Additionally, in GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning, the authors leverage generative 2D keypoint trajectories as an object-centric flow prior, abstracting away embodiment-specific details to guide reinforcement learning rewards effectively across different physical configurations.
The Mathematics of Coordinate-Free Control
To bridge the gap between high-level relative task specifications and low-level motor commands, the underlying control framework must natively support coordinate-free abstractions. This is where computational graphs and differential geometry play a vital role.
Cheng et al. introduce a systematic solution with RMPflow: A Computational Graph for Automatic Motion Policy Generation. By formulating local motion policies in a coordinate-free manner on a computational graph, RMPflow ensures that desired task-space dynamics remain perfectly consistent regardless of the robot's underlying degrees of freedom or kinematic structure.
Building on this foundation, Li et al. demonstrate in Multi-Objective Policy Generation for Multi-Robot Systems Using Riemannian Motion Policies that these coordinate-free control abstractions allow policies designed for simple point-mass models to be mapped directly onto heterogeneous robot teams. This mathematical decoupling ensures that the core task logic remains completely independent of specific hardware configurations, providing a rigorous foundation for cross-embodiment transfer.
Key Foundational Work
RMPflow: A Computational Graph for Automatic Motion Policy Generation (2018) — contributed 2 concept elements to this direction.
Multi-Objective Policy Generation for Multi-Robot Systems Using Riemannian Motion Policies (2019) — contributed 2 concept elements to this direction.
Graph-Structured Visual Imitation (2019) — contributed 1 concept elements to this direction.