Abstract
Learning diverse manipulation skills for real-world robots is severely bottlenecked by the reliance on costly and hard-to-scale teleoperated demonstrations. While human videos offer a scalable alternative, effectively transferring manipulation knowledge is fundamentally hindered by the significant morphological gap between human and robotic embodiments. To address this challenge and facilitate skill transfer from human to robot, we introduce Traj2Action, a novel framework that bridges this embodiment gap by using the 3D trajectory of the operational endpoint as a unified intermediate representation, and then transfers the manipulation knowledge embedded in this trajectory to the robot's actions. Our policy first learns to generate a coarse trajectory, which forms a high-level motion plan by leveraging both human and robot data. This plan then conditions the synthesis of precise, robot-specific actions within a co-denoising framework. Extensive real-world experiments on a Franka robot demonstrate that Traj2Action boosts performance by up to 27% and 22.25% over the $\pi_0$ baseline on short- and long-horizon tasks, respectively.
Introduction
The acquisition of complex robotic manipulation skills is currently limited by the high cost of data collection. Teleoperation, the standard method for gathering expert demonstrations, is time-consuming and difficult to scale. Conversely, human-centric video data is abundant but lacks the direct action labels (e.g., joint torques) required for robot control. This "morphological gap"—the physical difference between human hands and robotic grippers—makes direct skill transfer challenging.
Traj2Action addresses this by identifying the 3D trajectory of the operational endpoint as a "universal language" shared between embodiments. By focusing on the spatial path required to complete a task rather than the specific joint movements, the framework can leverage low-cost human data to improve robot performance.
Methodology
The Traj2Action architecture is a hierarchical, coarse-to-fine system built upon a Vision-Language-Action (VLA) backbone.
Unified Trajectory Representation
The framework defines a trajectory as a sequence of 3D Cartesian positions. For human data, this is the midpoint between the thumb and index finger; for robots, it is the end-effector position. This abstraction allows the model to learn task logic independently of the specific hardware.
Co-Denoising Framework
The system consists of two primary modules:
- Trajectory Expert ($g_T$): Trained on both human and robot data, this module predicts a high-level 3D motion plan based on visual input and language instructions.
- Action Expert ($\pi_\theta$): Trained exclusively on robot data, this module refines the coarse plan into precise, robot-specific actions, including rotation and gripper states.
Both experts are optimized jointly using a flow-matching objective, enabling them to denoise initial motion estimates into coherent action sequences during inference.
Experimental Setup
The framework was evaluated using a Franka Research 3 robot across four real-world tasks: picking a water bottle (PWB), picking a tomato and placing it in a tray (PTT), stacking rings (SRP), and stacking paper cups (SPC).
The dataset comprised 1,968 human demonstrations and 1,003 robot demonstrations. Notably, human data collection was found to be approximately 3.5 times faster than robot teleoperation. To align the different kinematic speeds of humans and robots, the researchers established a "3:1 sampling rule," where human trajectories are sampled at 30 FPS and robot trajectories at 10 FPS.
Results and Discussion
Traj2Action demonstrated significant performance gains over the $\pi_0$ baseline. In short-horizon tasks, it achieved a 27% improvement in success rate. In long-horizon tasks, it showed a 22.25% improvement in task progress.
Key findings include:
- Scaling Laws: Performance scales consistently with the volume of human data, suggesting that human demonstrations can effectively substitute for expensive robot data.
- Zero-Shot Generalization: The model showed a nascent ability to generalize to new task variations (e.g., placing objects in different colored trays) when exposed to human data of those variations, even if robot data for those specific variations was absent.
Conclusion
Traj2Action provides a scalable paradigm for human-to-robot skill transfer by bridging the morphological gap with 3D trajectories. While the framework excels at high-level planning, future work will focus on improving precision during contact-level interactions and extending the method to unconstrained "in-the-wild" human videos.
Summary updated