DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation

Xinyue Xu, Jieqiang Sun, Jing Dai, Siyuan Chen, Lanjie Ma, Ke Sun, Bin Zhao, Jianbo Yuan, Sheng Yi, Haohua Zhu, Yiwen LuarXiv 2025 · DOI · arXiv

Abstract

DexCanvas is a massive hybrid dataset for dexterous robot manipulation, featuring 7,000 hours of hand-object interactions. By utilizing a novel "real-to-sim" reinforcement learning pipeline, researchers transformed 70 hours of high-precision human motion-capture data into billions of frames of physically validated simulation. This approach provides the precise contact force profiles necessary for training advanced robotic hands, effectively bridging the gap between kinematic human demonstrations and the dynamic requirements of robotic control.

Introduction

Teaching robots to utilize anthropomorphic, human-like hands remains a primary challenge in robotics. While datasets for simple parallel-jaw grippers are abundant, dexterous manipulation involves complex interactions that are difficult to capture. Existing datasets typically suffer from one of two issues: real-world data (video or mocap) lacks accurate force measurements without intrusive sensors, while synthetic data often produces "floaty" or biomechanically impossible movements that fail to transfer to reality. Without understanding the exact contact forces and pressure distributions—the "feel" of manipulation—robots cannot master tasks ranging from gentle pinches to firm grips.

Methodology: The Real-to-Sim Pipeline

The core innovation of this work is a real-to-sim pipeline designed to extract physical ground truths from kinematic data.

Data Collection and RL Residual Control

The process began with recording 70 hours of human demonstrations using a 22-camera optical motion capture system and synchronized RGB-D sensors. These demonstrations covered 30 different objects, from simple primitives to complex household items.

To bridge the "kinematic gap," the researchers employed an RL Residual Controller within IsaacGym. Instead of merely replaying recorded motions, Reinforcement Learning (RL) policies were trained to control an actuated MANO hand model. This agent acts as a residual controller, applying subtle corrections to human joint angles to ensure the simulated object follows the recorded trajectory precisely while respecting physics.

The RL residual controller ensures physically consistent interactions in simulation, enabling force extraction.

Force Extraction and Augmentation

Because the simulation is physics-based, successful mimicry allows for the direct extraction of ground-truth contact forces and torques. This "physics as a validator" approach treats the simulator as a virtual force sensor. To scale the data, the team perturbed object sizes, initial positions, and material properties, expanding the 70-hour "seed" dataset into a 7,000-hour corpus—a 100x expansion.

The DexCanvas Dataset

DexCanvas represents a significant leap in scale and detail over previous benchmarks like GRAB or DexYCB.

  • Scale: 3.0 billion frames of data.
  • Taxonomy: Organized around 21 fundamental manipulation types (e.g., power grasps, precision pinches, in-hand rotations) based on the Cutkosky taxonomy.
  • Modality: Includes MANO hand parameters, 6-DoF object poses, multi-view RGB-D, and continuous, physically consistent contact force profiles.
DexCanvas captures diverse manipulations, such as rotating a cylinder, using the MANO hand model.

Key Results and Insights

The RL policies demonstrated high fidelity, achieving an 80.15% success rate in reproducing human demonstrations within the physics engine. The system also proved robust; even when object sizes were shifted by 20%, it maintained a 62.54% success rate, indicating that the generated synthetic variations are diverse and functional.

A key insight is that this data is morphology agnostic. Because it captures the underlying physics (forces and trajectories) rather than just joint angles, the data can be retargeted to various robot hand designs, from low-cost underactuated models to high-end research platforms. Furthermore, the dataset provides granular, per-finger force vectors, enabling the study of complex behaviors like "finger gaiting."

Conclusion and Future Work

While DexCanvas provides a robust foundation, limitations remain. Current efforts focus on rigid objects; expanding to deformable or articulated items (e.g., opening bags or using scissors) is a priority. Additionally, while the dataset includes RGB-D streams, the force reconstruction currently relies on motion capture. Future work aims to develop "vision-to-force" learning and unified "foundation" policies capable of handling all tasks and objects within a single model.

Summary updated

BibTeX
@misc{xu2025dexcanvas,
  author = {Xinyue Xu and Jieqiang Sun and Jing Dai and Siyuan Chen and Lanjie Ma and Ke Sun and Bin Zhao and Jianbo Yuan and Sheng Yi and Haohua Zhu and Yiwen Lu},
  title = {{DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation}},
  year = {2025},
  eprint = {2510.15786},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15786},
}
Figure