Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use

Haonan Chen, Cheng Zhu, Shuijing Liu, Yunzhu Li, Katherine Driggs-Campbell

TL;DR

Tool-as-Interface is a novel framework that allows robots to learn complex, dynamic tool-use skills directly from unconstrained human videos using just two standard RGB cameras. By framing the tool itself as the shared interface between human and robot, it bridges the embodiment and viewpoint gaps, achieving a 71% improvement in success rate over teleoperation while slashing data collection time by up to 77%.

What Problem Does This Paper Address?

Teaching robots to use diverse tools in dynamic environments is a holy grail for robotic manipulation. Traditionally, researchers rely on imitation learning via teleoperation (e.g., using a SpaceMouse or leader-follower setups like GELLO) or handheld grippers (like UMI). However, these methods are often slow, expensive, require specialized hardware, and introduce latency—making them poorly suited for high-speed, dynamic tasks like flipping an egg in a pan.

Learning directly from abundant human videos is a highly scalable alternative, but it introduces two massive hurdles:

  1. The Viewpoint Gap: Cameras capturing human videos are rarely in the exact same position as the robot's operational cameras.
  2. The Embodiment Gap: Human hands look and move completely differently than robotic grippers, creating a severe visual and kinematic mismatch.

This paper introduces a framework to bypass these gaps without requiring expensive hardware, tracking gloves, or robot-specific demonstrations.

Methods

The core philosophy of the paper is simple: treat the tool as the universal interface. Since both humans and robots manipulate the same physical tools to interact with the world, the framework focuses entirely on the tool's trajectory rather than the agent's anatomy.

Figure 1: Tool-as-Interface. We propose a scalable data collection and policy learning framework designed to transfer diverse, intuitive, and natural human data into effective visuomotor policies. The framework enables robots to learn robust policies that can operate effectively under challenging conditions, such as base and camera movement, and achieve high performance on a variety of complex manipulation tasks.

The pipeline consists of three main pillars:

  1. 3D-Aware View Augmentation: Human demonstrations are recorded using two uncalibrated RGB cameras. The authors use MASt3R (a foundation model for image matching) to reconstruct a global 3D point cloud of the scene. They then apply 3D Gaussian Splatting to synthesize novel viewpoints, artificially expanding the dataset and training the policy to be invariant to camera positions.
  2. Perception Alignment via Segmentation: To eliminate the embodiment gap, Grounded-SAM masks out human hands during training and the robot arm during deployment. This forces the visuomotor policy to focus exclusively on embodiment-agnostic features: the tool and the target objects.
  3. Tool-Centric Task-Space Actions: Instead of predicting joint angles or base-relative coordinates, the policy outputs the tool's 6D pose within a localized task frame (tracked via FoundationPose). Because actions are calculated relative to the workspace rather than the robot base, the policy remains functional even under severe external disturbances.

The final policy is trained using a standard Diffusion Policy architecture that maps single-view RGB images to these task-space tool actions.

Datasets

The authors evaluated their framework across five distinct, real-world manipulation tasks designed to test precision, speed, and gravity compensation:

  • Nail Hammering: Demands high-precision striking of a small target (<15.5 mm tip).
  • Meatball Scooping: Involves contact-sensitive manipulation of an unpredictable rolling object.
  • Pan Flipping: A high-speed, dynamic task evaluated using eggs, burger buns, and 3D-printed meat patties.
  • Wine Balancing: Requires gravity-aware insertion of a bottle into an unstable, free-moving rack.
  • Soccer Ball Kicking: Demands dynamic interception of a moving ball while avoiding obstacles.

All training data was collected purely from short human video clips (ranging from 50 seconds for pan flipping to 340 seconds for meatball scooping).

Key Insights and Ideas

  • Anatomy Doesn't Matter: By abstracting the action space to the tool's 6D pose, the robot doesn't need to understand human finger movements. It only needs to replicate what the tool did in the task space.
  • "Chicken-Head" Stabilization: Because the action representation is anchored to the task space, the robot naturally exhibits a stabilization effect when its base is shaken at low frequencies—similar to how a chicken keeps its head perfectly still while its body moves.
  • Unlocking the Un-teleoperable: Tasks requiring rapid acceleration and tactile intuition (like pan flipping) are notoriously difficult to demonstrate via teleoperation due to lag and lack of feedback. Human hands naturally possess these traits, and this framework successfully transfers that fluid dexterity to the robot.

Key Results

  • Superior Success Rates: The proposed method achieved a 71% higher success rate across tasks compared to diffusion policies trained on teleoperation data, which failed completely on highly dynamic tasks like pan flipping and soccer kicking.
  • Drastic Time Savings: Human video data collection reduced training data collection time by 77% compared to teleoperation and 41% compared to handheld grippers (UMI).
  • Extreme Robustness: The learned policies proved remarkably resilient to real-time physical disruptions. The robot successfully completed tasks despite active camera shaking, base shaking, and human interventions (such as moving the nail mid-strike or throwing extra meatballs into the pan).
  • Cross-Embodiment Generalization: A single policy trained on human videos was successfully deployed on both a 6-DOF UR5e and a 7-DOF Kinova Gen3 robot without any architecture modifications.

Limitations and Future Work

  • Perception Dependencies: The pipeline's accuracy hinges heavily on FoundationPose. If the 6D pose estimation fails or drifts during data preprocessing, it injects noise into the action labels, occasionally requiring data recollection.
  • Gaussian Splatting Artifacts: When synthesizing views that deviate too far from the original two camera angles, the generated images suffer from noise and reduced realism, which can degrade policy performance.
  • Rigid Tool Assumption: The framework assumes the tool is rigidly attached to the gripper and is non-deformable. It cannot currently accommodate flexible tools (like cables or brushes) or account for tool slippage inside the robot's physical gripper during high-impact contacts.

Summary updated

Abstract

Tool use is essential for enabling robots to perform complex real-world tasks, but learning such skills requires extensive datasets. While teleoperation is widely used, it is slow, delay-sensitive, and poorly suited for dynamic tasks. In contrast, human videos provide a natural way for data collection without specialized hardware, though they pose challenges on robot learning due to viewpoint variations and embodiment gaps. To address these challenges, we propose a framework that transfers tool-use knowledge from humans to robots. To improve the policy's robustness to viewpoint variations, we use two RGB cameras to reconstruct 3D scenes and apply Gaussian splatting for novel view synthesis. We reduce the embodiment gap using segmented observations and tool-centric, task-space actions to achieve embodiment-invariant visuomotor policy learning. We demonstrate our framework's effectiveness across a diverse suite of tool-use tasks, where our learned policy shows strong generalization and robustness to human perturbations, camera motion, and robot base movement. Our method achieves a 71\% improvement in task success over teleoperation-based diffusion policies and dramatically reduces data collection time by 77\% and 41\% compared to teleoperation and the state-of-the-art interface, respectively.
BibTeX
@inproceedings{chen2025tool,
  author = {Haonan Chen and Cheng Zhu and Shuijing Liu and Yunzhu Li and Katherine Driggs-Campbell},
  title = {{Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use}},
  booktitle = {Dex @ RSS},
  year = {2025},
  url = {https://openreview.net/forum?id=5udqMOeai1},
}
Figure