Leveraging Tactile Sensors for Low Latency Embedded Smart Hands for Prosthetic and Robotic Applications

Xiaying Wang, Fabian Geiger, Vlad Niculescu, Michele Magno, Luca Benini

Abstract

While computer vision has significantly advanced robotic perception, the sense of touch remains a bottleneck for dexterous manipulation. This paper presents SmartHand, an integrated hardware-software system designed for high-resolution, low-latency tactile sensing in robotic and prosthetic applications. By utilizing a 548-sensor tactile glove and a highly optimized convolutional neural network (CNN) deployed on an ARM Cortex-M7 microcontroller, the system achieves 100 frames per second (FPS) data throughput and object classification accuracy of 98.86%. We address the challenges of "TinyML" deployment, power efficiency, and the physical degradation of tactile sensors.

1. Introduction

In the quest to create robots that can interact naturally with the world, vision provides the "where," but touch provides the "how." Tactile sensing is essential for determining grip stability, surface texture, and the fragility of objects. However, existing tactile solutions often face a trade-off between resolution and latency, or require significant external computing power, making them unsuitable for portable prosthetics. SmartHand overcomes these barriers by integrating high-density sensing with on-device edge processing.

2. System Architecture

The SmartHand system is built upon three core pillars: hardware design, readout circuitry, and embedded machine learning.

2.1 Tactile Glove Hardware

The sensing layer consists of a flexible glove embedded with a Force Sensitive Resistor (FSR) matrix. A conductive polymer (Velostat) is sandwiched between 32x32 electrodes. While the grid allows for 1,024 potential "taxels" (tactile pixels), the design utilizes 548 active sensors distributed across the hand's surface to optimize coverage for grasping tasks.

2.2 High-Speed Readout Circuit

To process data from hundreds of sensors without lag, the researchers developed a custom analog front-end (AFE). A critical innovation is the electrical isolation scheme that prevents "cross-talk" between adjacent sensors. This architecture enables a scanning rate of 100 FPS, representing a 13.7× improvement over previous state-of-the-art tactile systems.

2.3 TinyML Implementation

Processing 100 FPS of tactile data locally requires extreme computational efficiency. The system employs a compact CNN based on the ResNet-18 architecture, optimized for an ARM Cortex-M7 microcontroller. By stripping the model to its essential components, the team reduced memory usage by an order of magnitude and decreased computations (MACCs) by 15.6× compared to standard models. Using the X-CUBE-AI expansion pack, the model achieves an inference time of 100ms.

3. Data and Methodology

The researchers introduced the ETHZ-STAG dataset to train and validate the system. This dataset contains 340,000 tactile frames covering 16 everyday objects (e.g., scissors, pens, cans) and an "empty hand" control class. To ensure the model's robustness, it was also benchmarked against the MIT-STAG dataset, demonstrating that the compact embedded model could match the performance of larger, PC-based architectures.

4. Results and Discussion

The SmartHand system demonstrates state-of-the-art performance across several metrics:

  • Classification Accuracy: The system achieved 98.86% top-1 accuracy on the ETHZ-STAG dataset using standard random splits.
  • Latency and Throughput: With 100 FPS throughput and 100ms inference latency, the system enables near-real-time reactive control.
  • Power Efficiency: The active power consumption is 505mW. On a 10% duty cycle, the system could operate for 20 hours on a small 1 Wh battery.

4.1 The Challenge of Sensor Degradation

A significant finding of this research is the impact of mechanical wear. Quantitative analysis showed that sensor response can diminish by up to 40% over time due to physical stress. This degradation directly impacts model generalization; while intra-session accuracy is high, inter-session accuracy (testing on different days) dropped to 77.84%. This highlights a critical hurdle for long-term deployment of tactile sensors in the field.

5. Conclusion

SmartHand represents a significant step toward truly "smart" prosthetics. By proving that high-resolution tactile sensing can be processed on low-power microcontrollers, the research paves the way for autonomous, reactive robotic hands. Future work will focus on adaptive learning algorithms to compensate for sensor wear and the integration of multi-modal data, such as IMU sensors, to further refine object interaction.

Summary updated

Abstract

Tactile sensing is a crucial perception mode for robots and human amputees in need of controlling a prosthetic device. Today, robotic and prosthetic systems are still missing the important feature of accurate tactile sensing. This lack is mainly due to the fact that the existing tactile technologies have limited spatial and temporal resolution and are either expensive or not scalable. In this article, we present the design and implementation of a hardware–software embedded system called SmartHand. It is specifically designed to enable the acquisition and real-time processing of high-resolution tactile information from a hand-shaped multisensor array for prosthetic and robotic applications. During data collection, our system can deliver a high throughput of 100 frames per second, which is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$13.7\times $ </tex-math></inline-formula> higher than previous related work. This has allowed the collection of a new tactile dataset consisting of 340 000 frames while interacting with 16 objects from everyday life during five different sessions. Together with the empty hand, the dataset presents a total of 17 classes. We propose a compact yet accurate convolutional neural network that requires one order of magnitude less memory and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$15.6\times $ </tex-math></inline-formula> fewer computations compared with related work without degrading classification accuracy. The top-1 and top-3 cross-validation accuracies on the collected dataset are, respectively, 98.86% and 99.83%. We further analyze the intersession variability and obtain the best top-3 leave-one-out-validation accuracy of 77.84%. We deploy the trained model on a high-performance ARM Cortex-M7 microcontroller achieving an inference time of only 100 ms minimizing the response latency. The overall measured power consumption is 505 mW. Finally, we fabricate a new control sensor and perform additional experiments to provide analyses on sensor degradation and slip detection. This work is a step forward in giving robotic and prosthetic devices a sense of touch by demonstrating the practicality of a smart embedded system that uses a scalable tactile sensor with embedded tiny machine learning.
BibTeX
@article{wang2022leveraging,
  author = {Xiaying Wang and Fabian Geiger and Vlad Niculescu and Michele Magno and Luca Benini},
  title = {{Leveraging Tactile Sensors for Low Latency Embedded Smart Hands for Prosthetic and Robotic Applications}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2022},
  doi = {10.1109/tim.2022.3165828},
}
Figure