Ground truth force distribution for learning-based tactile sensing: a finite element approach

Carmelo Sferrazza, Adam Wahlsten, Camill Trueeb, Raffaello D'Andrea

Abstract

Skin-like tactile sensors provide robots with rich feedback related to the force distribution applied to their soft surface. The complexity of interpreting raw tactile information has driven the use of machine learning algorithms to convert the sensory feedback to the quantities of interest. However, the lack of ground truth sources for the entire contact force distribution has mainly limited these techniques to the sole estimation of the total contact force and the contact center on the sensor's surface. The method presented in this article uses a finite element model to obtain ground truth data for the three-dimensional force distribution. The model is obtained with state-of-the-art material characterization methods and is evaluated in an indentation setup, where it shows high agreement with the measurements retrieved from a commercial force-torque sensor. The proposed technique is applied to a vision-based tactile sensor, which aims to reconstruct the contact force distribution purely from images. Thousands of images are matched to ground truth data and are used to train a neural network architecture, which is suitable for real-time predictions.

Introduction

Soft tactile sensors are essential for robots to interact safely with humans and fragile objects. Vision-based tactile sensors—which use internal cameras to track the deformation of a soft gel—are popular because they are inexpensive and high-resolution. However, mapping raw images to the actual forces applied to the sensor's surface is a significant challenge.

The physical vision-based tactile sensor prototype held in a hand.

While machine learning (ML) is the standard tool for this mapping, ML requires high-quality labels. In tactile sensing, it is relatively easy to measure the total force using a commercial sensor, but it is nearly impossible to measure the distribution of that force across the soft surface without placing bulky sensors between the object and the gel, which would fundamentally change the interaction. Consequently, most existing tactile sensors only estimate the total force or the center of contact, leaving the rich "shape" of the force distribution untapped.

Methodology

The authors bridge this gap by using a "simulation-to-real" pipeline to generate "unmeasurable" ground truth data. By pairing real-world sensor images with simulated 3D force distributions, they trained a deep neural network capable of reconstructing high-resolution contact forces in real-time.

Cross-sectional schematic of the sensor showing the gel layer and embedded markers used for tracking deformation.

1. Material Characterization and FEM Simulation

Extensive multiaxial stress tests (uniaxial, pure shear, and equibiaxial tension) were performed on the sensor's elastomers. This data was used to fit a non-linear, hyperelastic Ogden model, ensuring the simulation behaves exactly like the physical material. Using Abaqus, a high-fidelity 3D model of the sensor was built to simulate thousands of indentations and calculate nodal forces across the entire contact surface.

2. Data Labeling and Learning Architecture

The surface was discretized into bins, where forces from FEM nodes within each bin were summed to create a 3D force vector. The researchers collected 13,448 real-world images of indentations using a precise horizontal grid (0.55 mm spacing) on an automatic milling machine. For each image, they assigned the corresponding force distribution label from the FEM simulation. A feedforward Deep Neural Network (DNN) was then trained to map optical flow features (extracted from the images) to these 3D force distribution labels.

Key Results

  • High Accuracy: The total force predicted by the system showed excellent agreement with a hardware force-torque (F/T) sensor, with a Root-Mean-Square Error (RMSE) of roughly 0.06 N in the vertical direction.
  • Spatial Resolution: The system successfully reconstructed 3D force distributions across 400 bins, providing a spatial resolution of 1.6 mm, which is comparable to the human fingertip.
  • Real-Time Performance: The entire pipeline—from image capture to force prediction—runs at 40 Hz on a dual-core laptop, making it suitable for closed-loop robotic control.
  • Physics-Informed Labeling: By investing in material characterization, the authors turned a sensing problem into a simulation problem. The study demonstrates that hyperelastic models (like the Ogden model) are critical for matching real-world data at large deformations.

Conclusion and Future Work

This framework enables the training of high-resolution tactile sensors without the need for complex physical force-distribution measurement hardware. However, limitations remain. The current model was trained using a single spherical indenter; future work will need to include more diverse geometries (edges, cylinders) to generalize to arbitrary objects. Additionally, incorporating material aging and dynamic interaction effects (like friction during sliding) will be essential for long-term robotic applications.

Summary updated

Abstract

Skin-like tactile sensors provide robots with rich feedback related to the force distribution applied to their soft surface. The complexity of interpreting raw tactile information has driven the use of machine learning algorithms to convert the sensory feedback to the quantities of interest. However, the lack of ground truth sources for the entire contact force distribution has mainly limited these techniques to the sole estimation of the total contact force and the contact center on the sensor's surface. The method presented in this article uses a finite element model to obtain ground truth data for the three-dimensional force distribution. The model is obtained with state-of-the-art material characterization methods and is evaluated in an indentation setup, where it shows high agreement with the measurements retrieved from a commercial force-torque sensor. The proposed technique is applied to a vision-based tactile sensor, which aims to reconstruct the contact force distribution purely from images. Thousands of images are matched to ground truth data and are used to train a neural network architecture, which is suitable for real-time predictions.
BibTeX
@article{sferrazza2019ground,
  author = {Carmelo Sferrazza and Adam Wahlsten and Camill Trueeb and Raffaello D'Andrea},
  title = {{Ground truth force distribution for learning-based tactile sensing: a finite element approach}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2956882},
}

Loading related content...

Figure