Abstract
TrajSkill is a novel framework designed to bridge the "embodiment gap" in robotic learning. By utilizing sparse optical flow as a universal motion language, TrajSkill enables robots to learn manipulation skills directly from human videos without the need for paired human-robot datasets. The system extracts embodiment-agnostic trajectories from human demonstrations, generates robot-specific "mental rehearsal" videos via a Diffusion Transformer, and translates these visual plans into executable actions. Experimental results demonstrate significant improvements in visual fidelity and task success rates across both simulation and real-world benchmarks, achieving zero-shot imitation in complex manipulation tasks.
Introduction
The "embodiment gap" remains a primary hurdle in robotics: humans and robots possess vastly different physical structures, joint constraints, and appearances. Consequently, a robot cannot simply copy the pixel-level movements of a human hand. Traditional solutions often rely on expensive paired datasets or complex mathematical mappings to bridge this visual and morphological divide.
TrajSkill seeks to bypass these bottlenecks by identifying a representation of motion that is "embodiment-invariant." By focusing on the essential task dynamics rather than the specific morphology of the actor, the framework allows robots to interpret human intent and map it to their own physical capabilities.
The TrajSkill Framework
The TrajSkill pipeline operates in two primary stages to convert human demonstrations into robot actions.
1. Embodiment-Invariant Flow Sampling
The system first extracts dense optical flow from a human demonstration video. To strip away morphological details (like the appearance of skin or specific finger counts), it samples this flow into sparse trajectories. These trajectories represent the essential path of the motion—such as the arc of a hand reaching for a cup—while discarding the visual identity of the human mover.
2. Trajectory-Conditioned Video Generation
Using a Latent Diffusion Transformer (DiT) backbone, the model "imagines" how a robot would perform the observed motion. It takes the initial robot state, a text instruction (e.g., "Put the bread on the steak"), and the sparse trajectories as inputs to synthesize a temporally consistent video of the robot executing the task. This serves as a high-level visual plan.
3. Video-to-Action Policy
Finally, a learned policy $p(a|o, v)$ consumes the robot's real-time observation and the generated "plan" video to produce executable actions, specifically delta positions and rotations.
Experimental Evaluation
The authors evaluated TrajSkill across three diverse benchmarks: MetaWorld (50 simulated tasks), Franka Multi-Tasks (14 real-world tasks), and XSkill (a hybrid cross-embodiment dataset).
Key Results
- Visual Fidelity: On the MetaWorld benchmark, TrajSkill reduced the Frechet Video Distance (FVD) by 39.6% and Kernel Video Distance (KVD) by 36.6% compared to state-of-the-art video models, indicating significantly more realistic and physically consistent synthesis.
- Task Success: In cross-embodiment simulation tests, TrajSkill improved success rates by up to 16.7% over baselines like TinyVLA.
- Real-World Performance: In a "Put the Banana in the Basket" task, the system achieved a 90.9% success rate for picking and 81.8% for placing, outperforming generalist policies like OCTO in transfer settings.
Conclusion and Future Work
TrajSkill demonstrates that sparse optical flow can serve as an effective bridge between disparate embodiments. By treating video generation as a high-level planning step, the framework enables robots to "mentally rehearse" human skills before execution. Future work will focus on extending this approach to long-horizon, multi-stage tasks and exploring transfers between even more diverse morphologies, such as from humanoids to quadrupedal robots.
Summary updated