Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representation

Marina Y. Aoyama, João Moura, Namiko Saito, Sethu Vijayakumar

Introduction

Learning from Demonstration (LfD) has emerged as a powerful paradigm for teaching robots complex manipulation tasks. However, a significant bottleneck remains: generalization to objects with varying physical properties. A robot trained to wipe a surface with a soft sponge may fail when presented with a stiff or high-friction alternative, as the required force profiles and trajectories differ significantly. Traditional approaches to address this variability require exhaustive human demonstrations across the entire parameter space of object properties, which is often impractical. This paper presents a semi-supervised framework that enables robots to adapt to diverse physical properties using only a few human demonstrations by pre-training a haptic representation in simulation.

Methodology: Decoupling Sensing from Execution

The core innovation of this work is the decoupling of haptic perception from task execution. The framework consists of two primary components: a Haptic Representation Encoder and a Motion Generation Decoder.

Haptic Representation Encoder

The encoder is a Variational Auto-Encoder (VAE) designed to compress high-dimensional force-torque time-series data into a low-dimensional latent space. This model is trained using unsupervised data collected in the ROS-PyBullet simulator. The robot performs two types of exploratory actions—pressing and lateral motion—on virtual objects with varying stiffness and friction coefficients.

The robot performs pressing and lateral motion to characterize object properties during the exploration phase.

By training on 1,000 simulated trajectories, the encoder learns a latent representation that naturally organizes objects by their physical characteristics (e.g., grouping "stiff" vs. "compliant" objects) without explicit labels. The study revealed that "pressing" is optimized for identifying stiffness, while "lateral motion" is essential for surface friction.

Motion Generation Decoder and Few-Shot LfD

Once the haptic "sense" is pre-trained, a decoder is trained to map these latent representations to specific task trajectories. Unlike the encoder, the decoder is trained using a small set of human demonstrations (only 12 examples) on a physical KUKA iiwa robot. During the execution phase, the robot first performs a brief exploration to identify the object's latent representation, which then conditions the decoder to generate the appropriate wiping motion.

Overview of the task execution phase, showing the transition from exploration to task execution alongside force and position profiles.

Experimental Results

The proposed method was evaluated on a surface-wiping task using sponges of varying stiffness and friction. Key findings include:

  • Accuracy: The pre-trained model achieved a 50% reduction in Root Mean Squared Error (RMSE) for generated motions compared to a baseline LfD model without haptic pre-training.
  • Extrapolation: The framework demonstrated robust performance on objects with properties outside the range of the human demonstrations. For instance, in stiffness extrapolation, the proposed model maintained an RMSE of $6.4 \pm 3.6$, whereas the baseline error spiked to $25.1 \pm 22.9$.
  • Sim-to-Real Transfer: Despite being trained on simulated haptic data, the encoder successfully generalized to real-world objects, indicating that the fundamental physics of force interaction are well-captured in the latent space.

Conclusion

By leveraging unsupervised simulation data to build a foundational haptic representation, this research significantly reduces the human effort required for robot training. The ability to generalize from a handful of demonstrations to a wide array of physical object properties marks a significant step toward more adaptable and efficient robotic assistants. Future work will explore active perception strategies and online adaptation to dynamic object properties.

Summary updated

BibTeX
@inproceedings{aoyama2024few,
  author = {Marina Y. Aoyama and João Moura and Namiko Saito and Sethu Vijayakumar},
  title = {{Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10610502},
}
Figure