Ground Truth Force Distribution for Learning-Based Tactile Sensing: A Finite Element Approach

Carmelo Sferrazza, Adam Wahlsten, Camill Trueeb, Raffaello D'Andrea

Introduction

Giving robots a human-like sense of touch is a long-standing challenge in robotics. Soft, skin-like sensors are essential for delicate manipulation, but they are notoriously difficult to model due to their non-linear behavior. While vision-based tactile sensors—which use internal cameras to track the deformation of a soft gel—provide high-resolution data, translating these images into precise force values typically requires machine learning.

The primary bottleneck in this field is the lack of high-fidelity "ground truth" for distributed forces. While researchers can easily measure the total force applied to a sensor using commercial force-torque sensors, measuring the exact pressure distribution across every millimeter of the contact area is nearly impossible without interfering with the measurement itself. This paper introduces a novel pipeline that uses high-fidelity Finite Element Method (FEM) simulations to generate ground-truth labels for real-world camera images, enabling a neural network to reconstruct complex 3D contact force maps in real-time.

Sensor Architecture and Data Collection

The researchers developed a compact, vision-based tactile sensor designed for integration into robotic platforms. The sensor consists of a soft silicone gel layer embedded with fluorescent particles, which is monitored by an internal camera.

Cross-section diagram of the sensor showing the 4.5 mm gel layer embedded with particles and the overall 17 mm housing depth.

As shown in the cross-section below, the sensor's structure is carefully layered to facilitate optical tracking of deformation. The movement of the green particles within the gel provides the raw data needed to infer external forces.

Cross-section diagram of the sensor showing the 4.5 mm gel layer embedded with particles and the overall 17 mm housing depth.

To train their model, the authors used a high-precision milling machine to perform 13,448 controlled indentations on the sensor. For each indentation, they captured RGB images of the internal particle movement, creating a robust dataset of real-world deformations.

Methodology: Bridging Simulation and Reality

The core innovation of this work is using FEM not just for design, but as a high-resolution "virtual sensor" to label real-world data. The pipeline consists of four key steps:

  1. Material Characterization: The team performed physical tests (uniaxial tension, pure shear, and equibiaxial tension) on the sensor's silicone to build an accurate Ogden hyperelastic model. This captures how the material behaves under large deformations, which linear models fail to do.
  2. FEM Labeling: Every physical indentation was recreated in an FEM simulation. Because the simulation uses the accurate material model, it can calculate the 3D force distribution for every point on the surface—data impossible to measure physically.
  3. Feature Extraction: Instead of using raw pixels, the model extracts optical flow features, tracking how the internal particles move between frames.
  4. Learning Architecture: A feedforward Deep Neural Network (DNN) maps these optical flow movements to the 3D force distributions generated by the FEM.

Results and Validation

The system's performance was validated by comparing its predicted total force against a physical commercial force-torque (F/T) sensor. The results showed exceptional agreement, with a Root Mean Square Error (RMSE) on the vertical axis of only 0.06 N—matching the resolution limit of the physical sensor itself.

Key performance metrics include:

  • Real-Time Processing: The entire pipeline runs at 40 frames per second on a standard laptop CPU.
  • High Spatial Resolution: The system achieves a spatial resolution of 1.6 mm, which is comparable to the tactile resolution of a human fingertip.
  • Hyperelastic Accuracy: The study confirmed that non-linear hyperelastic modeling is essential; standard linear elastic simplifications resulted in significant errors during large deformations.

Conclusion and Future Work

By using FEM to generate high-resolution ground truth, the authors have unlocked the ability for robotic sensors to perceive detailed pressure maps rather than just simple point-contact forces. While the current model was trained primarily on vertical indentations with a spherical indenter, the methodology is generalizable. Future work will focus on expanding the dataset to include lateral sliding movements and more complex indenter geometries to further enhance the sensor's utility in complex manipulation tasks.

Summary updated

Abstract

Skin-like tactile sensors provide robots with rich feedback related to the force distribution applied to their soft surface. The complexity of interpreting raw tactile information has driven the use of machine learning algorithms to convert the sensory feedback to the quantities of interest. However, the lack of ground truth sources for the entire contact force distribution has mainly limited these techniques to the sole estimation of the total contact force and the contact center on the sensor's surface. The method presented in this article uses a finite element model to obtain ground truth data for the three-dimensional force distribution. The model is obtained with state-of-the-art material characterization methods and is evaluated in an indentation setup, where it shows high agreement with the measurements retrieved from a commercial force-torque sensor. The proposed technique is applied to a vision-based tactile sensor, which aims to reconstruct the contact force distribution purely from images. Thousands of images are matched to ground truth data and are used to train a neural network architecture, which is suitable for real-time predictions.
BibTeX
@article{sferrazza2019ground,
  author = {Carmelo Sferrazza and Adam Wahlsten and Camill Trueeb and Raffaello D'Andrea},
  title = {{Ground Truth Force Distribution for Learning-Based Tactile Sensing: A Finite Element Approach}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2956882},
}
Figure