Traj2Action: A Co-Denoising Framework for Trajectory-Guided Human-to-Robot Skill Transfer

Han Zhou, Jinjin Cao, Liyuan Ma, Xueji Fang, Guo-Jun Qi

Abstract

Learning diverse manipulation skills for real-world robots is severely bottlenecked by the reliance on costly and hard-to-scale teleoperated demonstrations. While human videos offer a scalable alternative, effectively transferring manipulation knowledge is fundamentally hindered by the significant morphological gap between human and robotic embodiments. To address this challenge and facilitate skill transfer from human to robot, we introduce Traj2Action, a novel framework that bridges this embodiment gap by using the 3D trajectory of the operational endpoint as a unified intermediate representation, and then transfers the manipulation knowledge embedded in this trajectory to the robot's actions. Our policy first learns to generate a coarse trajectory, which forms a high-level motion plan by leveraging both human and robot data. This plan then conditions the synthesis of precise, robot-specific actions within a co-denoising framework. Extensive real-world experiments on a Franka robot demonstrate that Traj2Action boosts performance by up to 27% and 22.25% over the $\pi_0$ baseline on short- and long-horizon tasks, respectively.

Introduction

The acquisition of complex robotic manipulation skills is currently limited by the high cost of data collection. Teleoperation, the standard method for gathering expert demonstrations, is time-consuming and difficult to scale. Conversely, human-centric video data is abundant but lacks the direct action labels (e.g., joint torques) required for robot control. This "morphological gap"—the physical difference between human hands and robotic grippers—makes direct skill transfer challenging.

Traj2Action addresses this by identifying the 3D trajectory of the operational endpoint as a "universal language" shared between embodiments. By focusing on the spatial path required to complete a task rather than the specific joint movements, the framework can leverage low-cost human data to improve robot performance.

Methodology

The Traj2Action architecture is a hierarchical, coarse-to-fine system built upon a Vision-Language-Action (VLA) backbone.

Overview of the Traj2Action framework, showing the interaction between the Trajectory Expert and the Action Expert.

Unified Trajectory Representation

The framework defines a trajectory as a sequence of 3D Cartesian positions. For human data, this is the midpoint between the thumb and index finger; for robots, it is the end-effector position. This abstraction allows the model to learn task logic independently of the specific hardware.

Co-Denoising Framework

The system consists of two primary modules:

  1. Trajectory Expert ($g_T$): Trained on both human and robot data, this module predicts a high-level 3D motion plan based on visual input and language instructions.
  2. Action Expert ($\pi_\theta$): Trained exclusively on robot data, this module refines the coarse plan into precise, robot-specific actions, including rotation and gripper states.

Both experts are optimized jointly using a flow-matching objective, enabling them to denoise initial motion estimates into coherent action sequences during inference.

Experimental Setup

The framework was evaluated using a Franka Research 3 robot across four real-world tasks: picking a water bottle (PWB), picking a tomato and placing it in a tray (PTT), stacking rings (SRP), and stacking paper cups (SPC).

The experimental environment, featuring multi-view camera setups (Top, Ego, Left) for both human and robot data collection.

The dataset comprised 1,968 human demonstrations and 1,003 robot demonstrations. Notably, human data collection was found to be approximately 3.5 times faster than robot teleoperation. To align the different kinematic speeds of humans and robots, the researchers established a "3:1 sampling rule," where human trajectories are sampled at 30 FPS and robot trajectories at 10 FPS.

Results and Discussion

Traj2Action demonstrated significant performance gains over the $\pi_0$ baseline. In short-horizon tasks, it achieved a 27% improvement in success rate. In long-horizon tasks, it showed a 22.25% improvement in task progress.

Key findings include:

  • Scaling Laws: Performance scales consistently with the volume of human data, suggesting that human demonstrations can effectively substitute for expensive robot data.
  • Zero-Shot Generalization: The model showed a nascent ability to generalize to new task variations (e.g., placing objects in different colored trays) when exposed to human data of those variations, even if robot data for those specific variations was absent.

Conclusion

Traj2Action provides a scalable paradigm for human-to-robot skill transfer by bridging the morphological gap with 3D trajectories. While the framework excels at high-level planning, future work will focus on improving precision during contact-level interactions and extending the method to unconstrained "in-the-wild" human videos.

Summary updated

Abstract

Learning diverse manipulation skills for real-world robots is severely bottlenecked by the reliance on costly and hard-to-scale teleoperated demonstrations. While human videos offer a scalable alternative, effectively transferring manipulation knowledge is fundamentally hindered by the significant morphological gap between human and robotic embodiments. To address this challenge and facilitate skill transfer from human to robot, we introduce Traj2Action, a novel framework that bridges this embodiment gap by using the 3D trajectory of the operational endpoint as a unified intermediate representation, and then transfers the manipulation knowledge embedded in this trajectory to the robot's actions. Our policy first learns to generate a coarse trajectory, which forms a high-level motion plan by leveraging both human and robot data. This plan then conditions the synthesis of precise, robot-specific actions (e.g., orientation and gripper state) within a co-denoising framework. Our work centers on two core objectives: first, the systematic verification of the Traj2Action framework's effectiveness-spanning architectural design, cross-task generalization, and data efficiency and second, the revelation of key laws that govern robot policy learning during the integration of human hand demonstration data. This research focus enables us to provide a scalable paradigm tailored to address human-to-robot skill transfer across morphological gaps. Extensive real-world experiments on a Franka robot demonstrate that Traj2Action boosts the performance by up to 27% and 22.25% over $π_0$ baseline on short- and long-horizon real-world tasks, and achieves significant gains as human data scales in robot policy learning.
BibTeX
@misc{zhou2025traj,
  author = {Han Zhou and Jinjin Cao and Liyuan Ma and Xueji Fang and Guo-Jun Qi},
  title = {{Traj2Action: A Co-Denoising Framework for Trajectory-Guided Human-to-Robot Skill Transfer}},
  year = {2025},
  eprint = {2510.00491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.00491},
}
Figure