From Human Hands to Robot Arms: Manipulation Skills Transfer via Trajectory Alignment

Han Zhou, Jinjin Cao, Liyuan Ma, Xueji Fang, Guo-Jun QiSubmission to ICLR 2026 · Withdrawn

TL;DR

Traj2Action enables robots to learn manipulation skills from fast, low-cost human hand videos by establishing a unified intermediate representation: the 3D trajectory of the operational endpoint. By combining a coarse trajectory planner trained on both human and robot data with a fine-grained robot action expert via a joint flow-matching framework, Traj2Action improves success and progress rates on short- and long-horizon tasks by over 20% while cutting expensive teleoperation time.

What Problem Does This Paper Address?

Imitation learning has driven remarkable progress in robotics, but modern Vision-Language-Action (VLA) models are bottlenecked by data collection. Gathering teleoperated robot demonstrations is notoriously expensive, slow, and requires skilled human operators. While human manipulation videos are abundant and easy to record, transferring skills from humans to robots faces a major hurdle: the morphological embodiment gap. A five-fingered human hand and a two-fingered parallel gripper interact with objects very differently. Direct joint retargeting or naive action alignment often fails because rotation semantics and contact dynamics do not match.

Methods

Traj2Action bridges this gap using a coarse-to-fine, dual-expert architecture built upon the $\pi_0$ VLA flow-matching backbone:

  • Unified 3D Trajectory Space: The authors identify the 3D Cartesian path of the "operational endpoint" as the greatest common divisor of manipulation. For humans, this endpoint is the midpoint between the thumb and index fingertips (extracted via MediaPipe keypoints fitted to the 3D MANO hand model); for the robot, it is the 3D position of the parallel gripper.
  • Dual-Expert Co-Denoising Policy:
  • Trajectory Expert ($g_T$): Initialized from the VLA action expert, it trains on both human and robot demonstrations ($\mathcal{D}_h \cup \mathcal{D}_r$) to predict a coarse $H$-step future 3D trajectory ($T_{t+1:t+H} \in \mathbb{R}^{H \times 3}$).
  • Action Expert ($\pi_\theta$): Trains strictly on robot data ($\mathcal{D}_r$) to output full 7-DoF robot actions (3D translation delta, 3D axis-angle rotation delta, and 1D gripper state). Crucially, its flow-matching denoising process is causally conditioned on the noisy trajectory produced by the Trajectory Expert. Both experts are denoised in parallel at inference with zero additional latency.
  • Visual & Kinematic Alignment: To minimize visual mismatch, human demonstrators wear a wrist-mounted camera mimicking the robot's egocentric wrist camera. To address kinematic speed discrepancies (humans move much faster than robot arms), human trajectories are sampled at 30 FPS while robot trajectories are sampled at 10 FPS, aligning their spatial step intervals.
Multi-view Images + Instruction
         │
         ▼
  [ VLM Backbone ]
         │
    ┌────┴─────────────────────────┐
    ▼                              ▼
[ Trajectory Expert ] ──causal──> [ Action Expert ]
(Human + Robot Data)   attention  (Robot Data Only)
High-level 3D Path                7-DoF Robot Control

Datasets

The authors collected real-world demonstrations across four diverse manipulation tasks using a Franka Research 3 arm equipped with a UMI parallel gripper:

  1. Pick up the water bottle (PWB): Basic pick-and-place (Short-Horizon).
  2. Pick up tomato and place into yellow/blue tray (PTT): Language grounding and goal routing (Short-Horizon).
  3. Stack rings on a pillar (SRP): Multi-step sequential placement requiring precision (Long-Horizon).
  4. Stack paper cups (SPC): Iterative stacking of deformable objects (Long-Horizon).

Across all tasks, the dataset contains 1,968 human demonstrations (totaling ~3.7 hours of collection) and 1,003 teleoperated robot demonstrations (~9.6 hours of collection).

Key Insights and Ideas

  • Abstracting to 3D Endpoints Bypasses Complex Morphologies: Previous works struggled to map human fingers to simple parallel grippers. Abstracting away finger articulation and focusing solely on the 3D operational path provides a clean, robust motion prior.
  • Naive Co-Training Fails: Simply feeding human keypoint actions directly into a shared action expert yielded only a marginal +2% gain over baseline. Traj2Action's trajectory intermediate space unlocked a +26% gain on the same task.
  • Human Data Collection is ~3.5× Faster: On average, recording human demonstrations took 5.09 seconds of execution per demo compared to 17.98 seconds via robot teleoperation. This efficiency gap grew to 3.85× on complex, multi-stage tasks.
  • Kinematic Resampling is Critical: Testing various sampling rates confirmed that a 3:1 ratio (30 FPS human vs. 10 FPS robot) gave the highest success rates (76% vs. 62% at 5 FPS human), demonstrating that temporal trajectory scale must match physical movement speed.

Key Results

  • Performance Gains: Compared to the fine-tuned $\pi_0$ baseline, Traj2Action achieved substantial improvements across all tasks:
  • Short-Horizon Success Rate: PWB increased from 48% to 76% (+28%); PTT increased from 50% to 76% (+26%).
  • Long-Horizon Task Progress: SRP increased from 23.75% to 44.75% (+21%); SPC jumped from 37.75% to 61.25% (+23.5%).
  • Data Substitution: Replacing robot data with human data saved substantial collection time without sacrificing performance: 240 human demos combined with reduced robot data achieved the same 60% success rate as a robot-only policy while reducing total data collection time by 20%.
  • Zero-Shot Task Transfer: When evaluated on an unseen prompt ("put tomato in blue tray") with robot training data containing only yellow-tray demos, Traj2Action achieved a 12% success rate (3/25 trials), whereas standard policies completely fail (0%).

Limitations and Future Work

  • Constrained Human Capture Setup: The current hand tracking system relies on a calibrated multi-camera setup with MediaPipe and MANO optimization, alongside a wrist-mounted ego camera. Extending this to uncalibrated, in-the-wild internet videos remains an open challenge.
  • Modest Zero-Shot Transfer: While a 12% zero-shot success rate proves that trajectory guidance can generalize beyond robot demonstrations, significant improvement is needed for reliable zero-shot task execution.
  • Orientation and Contact Nuances: The shared representation discards rotation and multi-finger articulation. Tasks heavily reliant on dexterous in-hand reorientation or precise contact forces still depend heavily on the robot demonstrations.

Summary updated

Abstract

Learning diverse manipulation skills for real-world robots is severely bottlenecked by the reliance on costly and hard-to-scale teleoperated demonstrations. While human videos offer a scalable alternative, effectively transferring manipulation knowledge is fundamentally hindered by the significant morphological gap between human and robotic embodiments. To address this challenge and facilitate skill transfer from human to robot, we introduce Traj2Action,a novel framework that bridges this embodiment gap by using the 3D trajectory of the operational endpoint as a unified intermediate representation, and then transfers the manipulation knowledge embedded in this trajectory to the robot's actions. Our policy first learns to generate a coarse trajectory, which forms an high-level motion plan by leveraging both human and robot data. This plan then conditions the synthesis of precise, robot-specific actions (e.g., orientation and gripper state) within a co-denoising framework. Extensive real-world experiments on a Franka robot demonstrate that Traj2Action boosts the performance by up to 27% and 22.25% over $\pi_0$ baseline on short- and long-horizon real-world tasks, and achieves significant gains as human data scales in robot policy learning.Our project website, featuring code and video demonstrations, is available at https://anonymous.4open.science/w/Traj2Action-4A45/.
BibTeX
@misc{zhou2025from,
  author = {Han Zhou and Jinjin Cao and Liyuan Ma and Xueji Fang and Guo-Jun Qi},
  title = {{From Human Hands to Robot Arms: Manipulation Skills Transfer via Trajectory Alignment}},
  year = {2025},
  eprint = {2510.00491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.00491},
}
Figure