Trajectory Conditioned Cross-embodiment Skill Transfer

Yuhang Tang, Yixuan Lou, Pengfei Han, Haoming Song, Xinyi Ye, Dong Wang, Bin ZhaoarXiv 2025 · DOI · arXiv

Abstract

TrajSkill is a novel framework designed to bridge the "embodiment gap" in robotic learning. By utilizing sparse optical flow as a universal motion language, TrajSkill enables robots to learn manipulation skills directly from human videos without the need for paired human-robot datasets. The system extracts embodiment-agnostic trajectories from human demonstrations, generates robot-specific "mental rehearsal" videos via a Diffusion Transformer, and translates these visual plans into executable actions. Experimental results demonstrate significant improvements in visual fidelity and task success rates across both simulation and real-world benchmarks, achieving zero-shot imitation in complex manipulation tasks.

Introduction

The "embodiment gap" remains a primary hurdle in robotics: humans and robots possess vastly different physical structures, joint constraints, and appearances. Consequently, a robot cannot simply copy the pixel-level movements of a human hand. Traditional solutions often rely on expensive paired datasets or complex mathematical mappings to bridge this visual and morphological divide.

TrajSkill seeks to bypass these bottlenecks by identifying a representation of motion that is "embodiment-invariant." By focusing on the essential task dynamics rather than the specific morphology of the actor, the framework allows robots to interpret human intent and map it to their own physical capabilities.

The TrajSkill Framework

The TrajSkill pipeline operates in two primary stages to convert human demonstrations into robot actions.

The two-stage TrajSkill pipeline for cross-embodiment skill transfer, showing the transition from human demonstration to robot execution.

1. Embodiment-Invariant Flow Sampling

The system first extracts dense optical flow from a human demonstration video. To strip away morphological details (like the appearance of skin or specific finger counts), it samples this flow into sparse trajectories. These trajectories represent the essential path of the motion—such as the arc of a hand reaching for a cup—while discarding the visual identity of the human mover.

2. Trajectory-Conditioned Video Generation

Using a Latent Diffusion Transformer (DiT) backbone, the model "imagines" how a robot would perform the observed motion. It takes the initial robot state, a text instruction (e.g., "Put the bread on the steak"), and the sparse trajectories as inputs to synthesize a temporally consistent video of the robot executing the task. This serves as a high-level visual plan.

3. Video-to-Action Policy

Finally, a learned policy $p(a|o, v)$ consumes the robot's real-time observation and the generated "plan" video to produce executable actions, specifically delta positions and rotations.

Detailed technical architecture of the TrajSkill framework, highlighting the integration of T5 language models, 3D VAE, and the Diffusion Transformer.

Experimental Evaluation

The authors evaluated TrajSkill across three diverse benchmarks: MetaWorld (50 simulated tasks), Franka Multi-Tasks (14 real-world tasks), and XSkill (a hybrid cross-embodiment dataset).

Key Results

  • Visual Fidelity: On the MetaWorld benchmark, TrajSkill reduced the Frechet Video Distance (FVD) by 39.6% and Kernel Video Distance (KVD) by 36.6% compared to state-of-the-art video models, indicating significantly more realistic and physically consistent synthesis.
  • Task Success: In cross-embodiment simulation tests, TrajSkill improved success rates by up to 16.7% over baselines like TinyVLA.
  • Real-World Performance: In a "Put the Banana in the Basket" task, the system achieved a 90.9% success rate for picking and 81.8% for placing, outperforming generalist policies like OCTO in transfer settings.

Conclusion and Future Work

TrajSkill demonstrates that sparse optical flow can serve as an effective bridge between disparate embodiments. By treating video generation as a high-level planning step, the framework enables robots to "mentally rehearse" human skills before execution. Future work will focus on extending this approach to long-horizon, multi-stage tasks and exploring transfers between even more diverse morphologies, such as from humanoids to quadrupedal robots.

Summary updated

BibTeX
@misc{tang2025trajectory,
  author = {Yuhang Tang and Yixuan Lou and Pengfei Han and Haoming Song and Xinyi Ye and Dong Wang and Bin Zhao},
  title = {{Trajectory Conditioned Cross-embodiment Skill Transfer}},
  year = {2025},
  eprint = {2510.07773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.07773},
}
Figure