RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot

Liang Heng, Xiaoqi Li, Shangqing Mao, Jiaming Liu, Ruolin Liu, Jingli Wei, Yu-Kai Wang, Yueru Jia, Chenyang Gu, Rui Zhao, Shanghang Zhang, Hao Dong

Abstract

Scaling robotic imitation learning is often hindered by the high cost and logistical complexity of collecting high-quality demonstrations. While human demonstrations are easily scalable, the visual domain gap between human hands and robotic grippers prevents direct policy transfer. We present RwoR (Robot without Robot), a framework that enables training robotic manipulation policies using only a wrist-mounted camera and human hand demonstrations. By employing a diffusion-based generative model to translate human hands into robotic grippers while maintaining environmental consistency, RwoR achieves performance comparable to state-of-the-art methods that require physical robotic hardware during data collection.

Introduction

Imitation learning has emerged as a primary paradigm for teaching robots complex manipulation tasks. However, the "data bottleneck" remains a significant hurdle. Traditional teleoperation requires expensive hardware and the physical presence of a robot, while human-centric data collection—though highly scalable—suffers from the "visual domain gap." A robot trained on human hands often fails to generalize when it perceives its own mechanical end-effector instead of the human skin and fingers seen during training.

RwoR addresses this challenge by providing a generative bridge. Instead of requiring a physical gripper or complex Augmented Reality (AR) setups, RwoR allows a user to record tasks with their bare hands. These recordings are then synthetically transformed into robot-compatible demonstrations, effectively decoupling data collection from robot hardware.

Methodology

The RwoR pipeline consists of three primary stages: data collection, action extraction, and visual translation.

Data Collection and Action Extraction

Data is collected using a GoPro Hero9 equipped with a fisheye lens, mounted on the human demonstrator's wrist to approximate a robot's eye-in-hand perspective. To extract the 6-Degrees-of-Freedom (6DoF) actions required for policy training, the system utilizes ORB-SLAM3 in conjunction with the GoPro’s internal Inertial Measurement Unit (IMU). This setup allows for precise tracking of the hand's pose in 3D space without the need for external motion capture systems.

The Generative Translation Pipeline

The core of RwoR is its ability to transform human hand videos into realistic robotic gripper videos. This is achieved through a diffusion-based image-to-image model, InstructPix2Pix (IP2P). To ensure the generated demonstrations are suitable for policy learning, the authors implemented several consistency measures:

  1. Temporal Cycle-Consistency (TCC): This aligns the timing between paired human and gripper demonstrations used during the training of the generative model.
  2. Segmentation and Inpainting: Using Segment Anything Model 2 (SAM2) and Inpaint Anything, the system isolates the hand and the manipulated object. This ensures that the background remains static during the translation process, preventing the policy from learning spurious correlations from background artifacts or hallucinations.

Experimental Evaluation

The researchers evaluated RwoR across a diverse set of 9 real-world tasks, including "Unplug Charger," "Wipe Plate," and "Table Rearrange."

Dataset Composition

The generative model was trained on a specialized dataset of 200 paired demonstrations (approximately 15,000 frames). In these pairs, a human performs a task with their hand and then replicates it as closely as possible using a handheld Universal Manipulation Interface (UMI) gripper. The dataset spans 25 object categories and 60 unique instances across various lighting conditions.

Comparative Results

RwoR was benchmarked against the UMI framework, which represents the state-of-the-art in handheld gripper data collection.

  • RwoR Success Rate: 78%
  • UMI Success Rate: 82%
  • Rule-Based Baseline: 37%

The results demonstrate that RwoR nearly matches the performance of hardware-heavy methods while significantly outperforming traditional rule-based image editing techniques. Furthermore, the model showed robust zero-shot generalization to unseen objects (e.g., novel colors) and novel actions, such as unstacking objects, which were not present in the paired training data.

Discussion and Limitations

While RwoR represents a significant step toward hardware-agnostic data collection, several challenges remain:

  • Temporal Jitter: Diffusion models can occasionally produce temporal inconsistencies, where the gripper's state (open/closed) may flicker or shift by a few frames. In precision tasks, these timing errors can lead to failed grasps.
  • Kinematic Constraints: Humans possess a greater range of motion and more degrees of freedom than most industrial robotic arms. If a human demonstrator performs a movement that exceeds the robot's joint limits, the resulting synthetic demonstration becomes physically un-executable for the target hardware.

Conclusion

RwoR demonstrates that high-fidelity robotic policies can be trained without the need for physical robots or specialized teleoperation rigs during the data collection phase. By leveraging generative AI to bridge the visual domain gap, RwoR paves the way for truly scalable robotic learning, where anyone with a consumer-grade camera can contribute to a robot's training data.

Summary updated

Abstract

Recent advancements in imitation learning have shown promising results in robotic manipulation, driven by the availability of high-quality training data. To improve data collection efficiency, some approaches focus on developing specialized teleoperation devices for robot control, while others directly use human hand demonstrations to obtain training data. However, the former requires both a robotic system and a skilled operator, limiting scalability, while the latter faces challenges in aligning the visual gap between human hand demonstrations and the deployed robot observations. To address this, we propose a human hand data collection system combined with our hand-to-gripper generative model, which translates human hand demonstrations into robot gripper demonstrations, effectively bridging the observation gap. Specifically, a GoPro fisheye camera is mounted on the human wrist to capture human hand demonstrations. We then train a generative model on a self-collected dataset of paired human hand and UMI gripper demonstrations, which have been processed using a tailored data pre-processing strategy to ensure alignment in both timestamps and observations. Therefore, given only human hand demonstrations, we are able to automatically extract the corresponding SE(3) actions and integrate them with high-quality generated robot demonstrations through our generation pipeline for training robotic policy model. In experiments, the robust manipulation performance demonstrates not only the quality of the generated robot demonstrations but also the efficiency and practicality of our data collection method. More demonstrations can be found at: https://rwor.github.io/
BibTeX
@inproceedings{heng2025rwor,
  author = {Liang Heng and Xiaoqi Li and Shangqing Mao and Jiaming Liu and Ruolin Liu and Jingli Wei and Yu-Kai Wang and Yueru Jia and Chenyang Gu and Rui Zhao and Shanghang Zhang and Hao Dong},
  title = {{RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246192},
}
Figure