Few-Shot Transfer of Tool-Use Skills Using Human Demonstrations With Proximity and Tactile Sensing

Marina Y. Aoyama, Sethu Vijayakumar, Tetsuya Narita

Introduction

This research, recently published in IEEE Robotics and Automation Letters (RA-L), addresses a fundamental challenge in robotics: enabling machines to use tools with the same grace and adaptability as humans without requiring weeks of task-specific training. The authors present a "few-shot" transfer framework that allows a robot to learn complex tool-use skills—such as painting, wiping, or sweeping—using as few as three human demonstrations. By combining simulation-based pre-training with multimodal sensing, the system effectively bridges the "sim-to-real" gap.

The "Double Contact" Challenge

Tool use is characterized as a "double contact" problem. To use a brush effectively, a robot must manage intrinsic contact (the interaction between its fingers and the tool handle) and extrinsic contact (the interaction between the tool tip and the environment). Modeling the physics of compliant tools like soft brushes or squishy sponges is notoriously difficult. While Learning from Demonstration (LfD) is a popular alternative, it typically requires massive datasets. Furthermore, policies trained in simulators often fail in the real world due to sensor noise and physical discrepancies.

Methodology: Two-Phase Skill Transfer

The researchers propose a two-phase framework to overcome these hurdles:

  1. Pre-training (Simulation): The robot is first trained on "primitive motions" in a MuJoCo simulator. These are basic behaviors like maintaining a specific downward force or following a surface. The architecture uses a Seq2Seq (LSTM) model where an encoder learns to extract contact features from sensor data, and a decoder generates the next action.
  2. Few-shot Fine-tuning (Real World): The model is transferred to a physical Franka Emika Panda arm. The "feature extraction" part of the network (the encoder) is frozen, while the final action layer (the decoder) is fine-tuned using only three human demonstrations performed via kinesthetic teaching.
The proposed two-phase framework: pre-training on primitive motions in simulation followed by few-shot fine-tuning with real-world human demonstrations.

Multimodal Sensing

The robot utilizes a sophisticated sensor suite to navigate its environment:

  • Tactile Sensors: 4x4 pressure grids on the fingertips sense shear force and slip.
  • Proximity Sensors: Six Time-of-Flight (ToF) sensors on the fingers look "down" to perceive environmental geometry before the tool makes contact, acting as an early warning system.

Experimental Setup and Datasets

The framework was evaluated across a diverse range of tools and environments to test its generalizability. The simulation dataset ($D_{prim}$) included 21 trajectories of primitive motions on inclined and stepped surfaces. In real-world trials, the robot was tested with a small brush, a wide brush, a sponge, and a broom on surfaces including stairs and even a deforming piece of paper.

Overview of the experimental components: (a) the sensorized gripper, (b) the variety of tools tested, and (c) the diverse simulated and real-world environments.

Key Insights and Results

  • Latent Space Organization: Using Principal Component Analysis (PCA), the researchers found that the LSTM encoder's latent space naturally organized itself by tool type and surface angle, even for tools it had never encountered before. This implicit disentanglement allows the model to generalize without explicit mathematical modeling of the tool.
  • Superior Force Control: The fine-tuned model maintained a target force of 0.5N with a mean error of only 0.11N, significantly outperforming models trained solely on human demonstrations (0.68N error).
  • Complex Geometry and Versatility: In a "Stairs" environment not seen during pre-training, the robot successfully wiped 59.2% of the surface area. It also successfully transitioned to a broom, sweeping 8 out of 10 dust pieces into a target area after just three demos.
  • Dynamic Adaptation: The system proved robust enough to follow the surface of a piece of paper even as a human hand pulled and deformed it during the task.

Conclusion and Future Work

This work demonstrates that freezing feature extraction layers and fine-tuning action layers can prevent overfitting and allow robots to learn from tiny sample sizes. While highly effective for surface-following tasks, future work will address limitations such as tool slippage within the gripper and the integration of global vision for long-range path planning in more complex environments.

Summary updated

BibTeX
@article{aoyama2025few,
  author = {Marina Y. Aoyama and Sethu Vijayakumar and Tetsuya Narita},
  title = {{Few-Shot Transfer of Tool-Use Skills Using Human Demonstrations With Proximity and Tactile Sensing}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  doi = {10.1109/lra.2025.3583608},
}
Figure