Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions

Carmelo Sferrazza, Raffaello D'Andrea

Abstract

Researchers from ETH Zurich have developed a novel framework to train vision-based tactile sensors entirely in simulation, eliminating the need for labor-intensive real-world data collection. By leveraging high-fidelity Finite Element Method (FEM) simulations, they generated synthetic training data to train a neural network that maps camera images of moving particles to 3D contact force distributions. The resulting model achieves real-time performance (120 Hz) on physical hardware with high accuracy, demonstrating effective zero-shot transfer from simulation to reality.

Introduction

Vision-based tactile sensors utilize internal cameras to monitor the deformation of a soft elastomer "skin," providing high-resolution feedback essential for sophisticated robotic manipulation. However, translating raw visual data into precise physical measurements—specifically 3D force distributions—remains a significant challenge. Soft materials like silicone exhibit non-linear, complex behaviors that are difficult to model analytically for real-time applications. Furthermore, training deep learning models requires massive labeled datasets, yet ground-truth 3D force distributions are nearly impossible to measure accurately at high resolution in the real world.

Methods: Bridging the Sim-to-Real Gap

The proposed pipeline addresses these challenges through a three-stage "sim-to-real" approach:

  1. High-Fidelity FEM Simulation: Using Abaqus, the researchers simulated 3,300 indentation trajectories across 21 distinct indenter shapes. This provided ground truth for both material deformation (particle displacement) and the corresponding 3D force distributions.
  2. Synthetic Image Generation: Simulated particle movements were projected into a virtual pinhole camera model. To enhance robustness, the authors introduced random RGB coloring and noise, mimicking real-world sensor variability.
  3. Learning Architecture: Force estimation was framed as an image-to-image translation task. A lightweight U-Net architecture maps 88x88 input features to a 20x20x3 matrix representing the $F_x, F_y,$ and $F_z$ components of the force distribution.
  4. Real-World Calibration: To ensure the sim-trained model functions on physical hardware, a calibration step remaps real-world images—accounting for lens distortion and refraction—into the ideal pinhole coordinates used during training.
Comparison of simulated and real-world indentation setups and the resulting sensor images used for training and validation.

Datasets

The study utilized two primary datasets to train and validate the model:

  • Training Set (Simulated): Comprised of 3,300 trajectories (50 steps each) using 21 different indenter shapes, including flat, round, and sharp profiles. This diversity ensured the model encountered a wide range of shear and pressure conditions.
  • Test Set (Real-World): Consisted of 1,100 data points collected using a high-precision milling machine integrated with a commercial force/torque sensor. Validation included six indenters, some of which were novel shapes not seen during the simulation phase.
The variety of indenter shapes used in the Finite Element Method (FEM) simulations to generate a diverse training dataset.

Key Insights and Results

A pivotal finding was that training on raw subsampled images outperformed dense optical flow. While optical flow algorithms often assume constant particle density, real-world high-pressure contact causes particles to spread radially—a feature captured more effectively by raw image data. By representing 3D force as a three-channel "image," the researchers leveraged convolutional neural networks' spatial reasoning capabilities.

The model achieved a Root Mean Squared Error (RMSE) between 0.005 N and 0.014 N for individual force components on real-world tests. Operating at 120 Hz on a standard laptop CPU, the pipeline is twice as fast as previous optical-flow-based methods. Notably, the model demonstrated strong generalization, accurately reconstructing force distributions for multi-contact scenarios and complex objects like a tape dispenser, despite being trained only on single-object indentations.

Conclusion and Future Work

While the zero-shot transfer is successful, a small "sim-to-real gap" persists, with higher accuracy observed in simulated environments. Future research may explore explicit domain adaptation to further close this gap. Additionally, while the inference is fast, the FEM data generation is computationally intensive. Exploring faster simulation techniques or accounting for time-dependent material properties like hysteresis could further enhance the capabilities of high-resolution tactile sensing in robotics.

Summary updated

BibTeX
@inproceedings{sferrazza2021sim,
  author = {Carmelo Sferrazza and Raffaello D'Andrea},
  title = {{Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions}},
  booktitle = {Soft},
  year = {2021},
  doi = {10.1089/soro.2020.0213},
}
Figure