Relative Task Space Representations for Cross Embodiment Transfer

21 concepts15 papers1 proposal

Traditional imitation learning and reinforcement learning often define robotic policies directly within the robot's joint space or relative to a fixed robot-base frame. While straightforward, this approach tightly couples the learned behavior to a specific hardware configuration. If you change the link lengths, add a joint, or attempt to learn from a human video demonstration, the policy fails due to the embodiment gap.

To bypass this limitation, a powerful paradigm has emerged: shifting the focus from how the robot moves to how the environment changes. By framing states and actions in relative, coordinate-free task spaces, policies become inherently embodiment-agnostic.

The Power of Relative Invariance

Why does this shift work? Geometry is invariant, whereas kinematics are platform-specific. If a task is defined as moving an end-effector relative to an object rather than moving joints relative to a base, the underlying mechanical structure of the robot becomes irrelevant.

For instance, Wilcox et al. show in Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learning that transforming visual observations directly into an end-effector-centric coordinate frame makes the policy invariant to the robot arm's specific joint configurations. This mathematical transformation enables zero-shot transfer across entirely different physical platforms. Similarly, Ramesh et al. demonstrate in Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies that formulating policies in the $SE(3)$ pose space rather than joint coordinates abstracts the task logic away from hardware kinematics, facilitating seamless cross-platform deployment.

Object-Centric Trajectories and Task Graphs

When the robot's physical structure is abstracted away, object trajectories and spatial relationships serve as a universal intermediate language. If a policy specifies how an object should move, any robot can execute that trajectory using standard inverse kinematics (IK).

Argus proposes in DITTO: Demonstration Imitation by Trajectory Transformation that manipulation tasks can be represented purely as a sequence of 3D object poses relative to the environment. By focusing entirely on the object's path rather than human skeletal movements or robot joint angles, the system completely sidesteps the human-to-robot correspondence gap. To scale this across complex scenes, Zeng et al. develop GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes, which synthesizes 6-DOF object trajectories that can be mapped onto any robotic embodiment.

[Human Video] ──> [Extract Object-Centric Graph/Trajectory] ──> [Inverse Kinematics] ──> [Any Robot Joint Space]

When tasks involve intricate geometries or multiple objects, a single global object pose is insufficient. Instead, the scene can be modeled as a structured graph of relative spatial constraints:

Scaling to Bimanual Coordination and Dexterity

The benefits of relative task-space representations become even more pronounced in bimanual manipulation and dexterous control, where coordinating high-dimensional joint spaces is notoriously difficult. By parameterizing actions as the relative poses between hands and objects, complex coordination patterns become highly portable.

In Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks, Gao et al. extract embodiment-independent hand-object relationships to ensure skills remain robust to workspace clutter. Similarly, Zhou et al. demonstrate in One-Shot Real-World Demonstration Synthesis for Scalable Bimanual Manipulation and BiNoMaP: Learning Category-Level Bimanual Non-Prehensile Manipulation Primitives that training policies using 6-DoF end-effector poses relative to object frames—rather than joint coordinates—enables zero-shot transfer across entirely different dual-arm hardware configurations. To anchor these skills to novel objects without retraining, Zhou et al. also propose VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Bimanual Robotic Manipulation, using vision-language models to ground relative spatial relationships across different robot bodies.

This abstraction extends down to fine-grained dexterous manipulation. Tao et al. show in DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies that tracking fingertip trajectories relative to a palm-centric frame allows dexterous manipulation policies to transfer across distinct robotic hand architectures. Additionally, in GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning, the authors leverage generative 2D keypoint trajectories as an object-centric flow prior, abstracting away embodiment-specific details to guide reinforcement learning rewards effectively across different physical configurations.

The Mathematics of Coordinate-Free Control

To bridge the gap between high-level relative task specifications and low-level motor commands, the underlying control framework must natively support coordinate-free abstractions. This is where computational graphs and differential geometry play a vital role.

Cheng et al. introduce a systematic solution with RMPflow: A Computational Graph for Automatic Motion Policy Generation. By formulating local motion policies in a coordinate-free manner on a computational graph, RMPflow ensures that desired task-space dynamics remain perfectly consistent regardless of the robot's underlying degrees of freedom or kinematic structure.

Building on this foundation, Li et al. demonstrate in Multi-Objective Policy Generation for Multi-Robot Systems Using Riemannian Motion Policies that these coordinate-free control abstractions allow policies designed for simple point-mass models to be mapped directly onto heterogeneous robot teams. This mathematical decoupling ensures that the core task logic remains completely independent of specific hardware configurations, providing a rigorous foundation for cross-embodiment transfer.

Key Foundational Work

RMPflow: A Computational Graph for Automatic Motion Policy Generation (2018) — contributed 2 concept elements to this direction.
Multi-Objective Policy Generation for Multi-Robot Systems Using Riemannian Motion Policies (2019) — contributed 2 concept elements to this direction.
Graph-Structured Visual Imitation (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_object_centric_task_representation_763d7310,
  title = {object-centric task representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-task-representation/rdr_763d7310eaed40c483d4638add32e127},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_object_centric_task_representation_763d7310_v1,
  title = {object-centric task representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-task-representation/rdr_763d7310eaed40c483d4638add32e127/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake