Tool-as-Interface: Learning Robot Tool Use from Human Play through Imitation Learning

Haonan Chen, Cheng Zhu, Yunzhu Li, Katherine Rose Driggs-Campbell

TL;DR

Teaching robots complex tool use via teleoperation or specialized handheld grippers is often slow, hardware-dependent, and prone to failure on fast, dynamic actions. Tool-as-Interface bypasses these bottlenecks by having humans naturally manipulate tools while recorded by two standard RGB cameras, using 3D reconstruction, novel-view augmentation, and embodiment masking to train robust visuomotor diffusion policies. The system slashes data collection time by up to 77% while conquering high-speed, dynamic tasks—like pan-flipping and bottle balancing—that teleoperation cannot solve.

What Problem Does This Paper Address?

While robotic manipulation has made major strides in simple parallel-jaw grasping, dexterous tool use remains extraordinarily difficult. Current imitation learning pipelines depend on two common data collection paradigms:

  1. Teleoperation (e.g., SpaceMouse, GELLO, body retargeting): Slow, introduces latency, and requires constant access to robot hardware. High-speed, dynamic, or contact-rich tasks are almost impossible to teleoperate.
  2. Handheld Grippers (e.g., UMI): Untethers collection from robot arms, but requires 3D printing, custom hardware, and robust visual SLAM. Moreover, tools can slip under inertial forces or during contact-heavy manipulation, breaking tracking.

Humans, however, naturally use tools with speed, fluidity, and compliancy ("human play"). This paper asks: Can we train robot policies directly from unstructured human play by letting the tool itself serve as the shared interface between human and robot?

Methods

The authors introduce an end-to-end framework translating dual-camera human play into robot visuomotor policies:

[Two RGB Cameras] 
       │
       ├──► MASt3R (Extrinsics-free 3D Point Cloud) ──► 3D Gaussian Splatting (Novel View Augmentation)
       ├──► Grounded-SAM (Masks hands in train, robot in test) ──► Embodiment-Agnostic Observations
       └──► FoundationPose (Extracts 6D Tool Pose) ──► Task-Space Tool Action: T_tool^task
                                                                    │
                                                         [Visuomotor Diffusion Policy]
  • Embodiment-Agnostic Perception: To bridge the morphological gap between a human hand and a robot gripper, the pipeline uses Grounded-SAM to segment and mask out human hands during training demonstrations, and masks the robot end-effector during deployment. The policy only "sees" the tool and the surrounding scene.
  • 3D Perception & Novel-View Augmentation: Rather than requiring depth sensors or calibrated camera rigs, the system runs MASt3R on pairs of RGB images to build 3D representations without camera intrinsics or extrinsics. It then fits 3D Gaussian Splatting (3DGS) to render novel viewpoints, training the policy to be invariant to camera shifts.
  • Tool-Centric Action Representation: FoundationPose tracks the 6D pose of the tool in the camera frame, which is converted to an invariant task space ($T_{\text{tool}}^{\text{task}}$). At test time, a Diffusion Policy predicts the tool trajectory, which is algebraically mapped into robot end-effector commands ($T_{\text{eef}}^{\text{base}}$), compensating for base motion.

Datasets

The authors evaluated their pipeline on five real-world benchmarks demanding diverse capabilities:

  1. Nail Hammering: High precision and contact force.
  2. Meatball Scooping: Extrinsic dexterity, contact-rich manipulation.
  3. Pan Flipping (Toy egg, 3D-printed patty, real burger bun): Extreme dynamics, gravity-overcoming, extrinsic dexterity.
  4. Wine Bottle Balancing: Precision and compliance under contact.
  5. Soccer Ball Kicking: Rapid dynamic swing and trajectory control.

Key Insights and Ideas

  • The Tool as an Embodiment Bridge: Prior cross-embodiment approaches struggle to match human hand kinematics to robot grippers. Focusing purely on the tool’s 6D trajectory abstracts away kinematic differences, as both human hands and robotic grippers manipulate the exact same tool.
  • Dynamic Motions Break Existing Setups: High-acceleration maneuvers like pan-flipping cause tools clamped in handheld grippers (like UMI) to experience inertial slippage. Human hands naturally grip and adapt, making raw human demonstrations far superior for dynamic tasks.
  • SLAM-Free RGB Reconstruction: Handheld setups often fail when feature tracking degrades (e.g., textureless tables or rapid swings). Generating uncalibrated 3D representations via MASt3R and 3D Gaussian Splatting creates rich view-augmented training data without relying on real-time SLAM.

Key Results

  • Higher Success Rates: Compared to diffusion policies trained on teleoperation data, Tool-as-Interface achieved an average 71% improvement in success rate. Dynamic tasks (pan flipping, wine balancing, soccer kicking) were completely infeasible via teleoperation, yet the proposed method achieved high success (e.g., 12/12 on egg flipping, 10/12 on patty flipping, 8/10 on bottle balancing).
  • Massive Time Savings: The method reduced demonstration collection time by 77% compared to teleoperation and 41% compared to UMI.
  • Comparison with UMI: On nail hammering, UMI achieved 0/13 success with 25 demonstrations (~180s) and required 100 demonstrations (~720s) to hit 13/13. Tool-as-Interface achieved 13/13 with only 40 demonstrations (~180s).
  • Generalization:
  • Objects: The pan-flipping policy generalized across a toy egg, a 3D-printed patty, and a soft burger bun using only 13 demonstrations.
  • Tools: Evaluated on 5 pan sizes and shapes, policies trained on large, medium, and square pans achieved 75% and 83% success on large/medium pans, though success dropped on smaller or shallow-edged square pans.

Limitations and Future Work

  • Geometric Limits on Generalization: While the policy generalizes well across similar tools, radical geometry shifts (e.g., tiny pans with reduced surface area or square pans with low sidewalls) cause significant performance drops (down to 8–17%).
  • Pose Estimation Reliance: Action labeling depends on FoundationPose; tools with severe occlusions, extreme symmetry, or lack of 3D CAD priors could degrade 6D pose tracking.
  • Tool-Affixed Manipulation: The system currently assumes rigid attachment between the robot end-effector and the tool during deployment, leaving unconstrained tool grasping/re-grasping for future exploration.

Summary updated

Abstract

Tool use is critical for enabling robots to perform complex real-world tasks, and leveraging human tool-use data can be instrumental for teaching robots. However, existing data collection methods like teleoperation are slow, prone to control delays, and unsuitable for dynamic tasks. In contrast, human play—where humans directly perform tasks with tools—offers natural, unstructured interactions that are both efficient and easy to collect. Building on the insight that humans and robots can share the same tools, we propose a framework to transfer tool-use knowledge from human play to robots. Using two RGB cameras, our method generates 3D reconstruction, applies Gaussian splatting for novel view augmentation, employs segmentation models to extract embodiment-agnostic observations, and leverages task-space tool-action representations to train visuomotor policies. We validate our approach on diverse real-world tasks, including meatball scooping, pan flipping, wine bottle balancing, and other complex tasks. Our method achieves a 71\% higher average success rate compared to diffusion policies trained with teleoperation data and reduces data collection time by 77\%, with some tasks solvable only by our framework. Compared to hand-held gripper, UMI~\cite{chi2024universal}, our method cuts data collection time by 41\%. Additionally, our method bridges the embodiment gap, improves robustness to variations in camera viewpoints and robot configurations, and generalizes effectively across objects and spatial setups.
BibTeX
@inproceedings{chen2025tool,
  author = {Haonan Chen and Cheng Zhu and Yunzhu Li and Katherine Rose Driggs-Campbell},
  title = {{Tool-as-Interface: Learning Robot Tool Use from Human Play through Imitation Learning}},
  booktitle = {Dex @ RSS},
  year = {2025},
  url = {https://openreview.net/forum?id=5udqMOeai1},
}
Figure