Deep Learning Classification of Touch Gestures Using Distributed Normal and Shear Force

Hojung Choi, Dane Brouwer, Michael A. Lin, Kyle T. Yoshida, Carine Rognon, Benjamin Stephens-Fripp, Allison M. Okamura, Mark R. Cutkosky

Abstract

When humans socially interact with another agent (e.g., human, pet, or robot) through touch, they do so by applying varying amounts of force with different directions, locations, contact areas, and durations. While previous work on touch gesture recognition has focused on the spatio-temporal distribution of normal forces, we hypothesize that the addition of shear forces will permit more reliable classification. We present a soft, flexible skin with an array of tri-axial tactile sensors for the arm of a person or robot. We use it to collect data on 13 touch gesture classes through user studies and train a Convolutional Neural Network (CNN) to learn spatio-temporal features from the recorded data. The network achieved a recognition accuracy of 74% with normal and shear data, compared to 66% using only normal force data. Adding distributed shear data improved classification accuracy for 11 out of 13 touch gesture classes.

Introduction

Social touch is a fundamental component of interaction between humans, pets, and increasingly, robots. While humans naturally perceive a wide range of tactile sensations, current robotic skins primarily focus on normal force (pressure). This creates a significant "sensory gap," as many social gestures—such as stroking, rubbing, or pinching—rely heavily on lateral movements and skin stretch. Without measuring these shear forces, machines struggle to distinguish between gestures with similar pressure profiles but distinct lateral components. This research presents a soft, flexible electronic skin capable of 3D force sensing and a deep learning framework to classify complex social touch gestures.

Hardware Design and Data Collection

The core of the system is a novel tri-axial magnetic tactile sensor array. The hardware consists of a 5×9 grid of "taxels" (tactile pixels) integrated into a soft polydimethylsiloxane (PDMS) structure. Each taxel contains a permanent magnet suspended above a Hall Effect sensor on a flexible printed circuit board (FPCB).

The tri-axial tactile sensor array architecture, featuring a 5x9 grid of taxels embedded in a soft PDMS structure.

When the skin is deformed by touch, the magnets move relative to the sensors, allowing for the detection of magnetic flux changes across the x, y, and z axes. The PDMS structure utilizes a hollow-beam design specifically engineered to maximize lateral translation (shear) while minimizing parasitic rotation, ensuring high-fidelity signal capture.

To train the classification model, the researchers conducted a user study with 11 participants. The sensor was wrapped around a mannequin's forearm to simulate a realistic interaction scenario. Participants performed 13 distinct social touch gestures (e.g., Stroke, Scratch, Poke, Pinch), resulting in a dataset of 3,861 five-second recordings.

Deep Learning Approach

The tactile data is treated as a spatio-temporal "video" where each frame represents the 3D force distribution across the grid. A Convolutional Neural Network (CNN) was employed to automatically learn features from this high-dimensional data, bypassing the need for hand-coded heuristics like peak force or duration. The model takes a 3D "image stack" of force data and learns to recognize the unique spatio-temporal signatures of different gestures.

Results and Analysis

The inclusion of shear force data significantly improved classification performance. The CNN achieved an overall accuracy of 74% when utilizing both normal and shear data, compared to 66% when restricted to normal force alone.

User study setup and visualization of the spatio-temporal force data captured during a touch gesture.

Shear data was particularly effective for gestures with high lateral content, such as slaps and strokes, and for distinguishing between gestures like "pinch" and "poke." In a pinch, shear vectors point toward each other, a feature entirely invisible to pressure-only sensors. Adding distributed shear data improved classification accuracy for 11 out of 13 touch gesture classes, proving that lateral information is a critical differentiator in social touch.

Conclusion and Future Work

While the results are promising, challenges remain. Significant variability exists in how different individuals perform gestures; for instance, one user's "pat" might be more forceful than another's "slap." Future work will explore more advanced temporal modeling architectures, such as LSTMs or Transformers, to better capture long-term gesture dynamics. Additionally, the team aims to scale the sensor array into full-limb "sleeves" to enable more comprehensive human-robot interaction and real-time response capabilities.

Summary updated

Abstract

When humans socially interact with another agent (e.g., human, pet, or robot) through touch, they do so by applying varying amounts of force with different directions, locations, contact areas, and durations. While previous work on touch gesture recognition has focused on the spatio-temporal distribution of normal forces, we hypothesize that the addition of shear forces will permit more reliable classification. We present a soft, flexible skin with an array of tri-axial tactile sensors for the arm of a person or robot. We use it to collect data on 13 touch gesture classes through user studies and train a Convolutional Neural Network (CNN) to learn spatio-temporal features from the recorded data. The network achieved a recognition accuracy of 74% with normal and shear data, compared to 66% using only normal force data. Adding distributed shear data improved classification accuracy for 11 out of 13 touch gesture classes.
BibTeX
@inproceedings{choi2022deep,
  author = {Hojung Choi and Dane Brouwer and Michael A. Lin and Kyle T. Yoshida and Carine Rognon and Benjamin Stephens-Fripp and Allison M. Okamura and Mark R. Cutkosky},
  title = {{Deep Learning Classification of Touch Gestures Using Distributed Normal and Shear Force}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981457},
}
Figure