Crossmodal Pattern Discrimination in Humans and Robots: A Visuo-Tactile Case Study

Focko L. Higgen, Philipp Ruppel, Michael Görner, Matthias Kerzel, Norman Hendrich, Jan-Frederik Feldheim, Stefan Wermter, Jianwei Zhang, Christian Gerloff

Abstract

This study investigates the mechanisms of crossmodal sensory integration by comparing the performance of younger humans, older humans, and artificial neural networks (ANNs) in a visuo-tactile pattern discrimination task. Utilizing a robotic platform equipped with advanced tactile sensors, the researchers evaluated two distinct AI architectures: "late integration" (V-architecture) and "early integration" (Y-architecture). The results demonstrate that early integration—combining high-level sensory features before final classification—significantly enhances performance. While the AI models outperformed older humans, younger humans maintained a superior integration efficiency, suggesting that age-related cognitive decline may stem from failing integration mechanisms rather than simple sensory degradation.

Introduction

As humans age, the ability to process "crossmodal" information—such as matching the tactile feel of a surface to its visual appearance—often declines. A fundamental question in neuroscience is whether this impairment results from the degradation of individual senses (unimodal) or a breakdown in the brain's ability to combine those senses (integration). This study bridges neuroscience and robotics by using AI as a computational model to explore these integration strategies and determine if state-of-the-art robotic sensors can replicate human-level acuity.

Methodology

Experimental Setup

The researchers employed a Shadow C6 Dexterous Hand fitted with BioTac tactile sensors to perform a pattern discrimination task. The objective was to determine if a Braille pattern felt by the robotic finger matched a visual pattern displayed on a screen. To ensure a valid comparison between biological and artificial subjects, an adaptive staircase procedure was used to equalize difficulty. This adjusted stimulus intensity (pin height for touch and contrast for vision) so that all subjects—human and robot—began with a baseline unimodal accuracy of approximately 80%.

Experimental setup showing (A) the human participant environment, (B) the visual stimulus with Perlin noise, and (C) the robotic Shadow Hand with BioTac sensors.

Stimuli and Data Collection

The task utilized four specific 4-dot Braille patterns. Tactile data consisted of 3,000 samples from the BioTac sensors, capturing 20 channels of pressure and impedance data over 150 time-steps. Visual data involved synthetic images where the target patterns were superimposed on "Perlin noise" backgrounds to simulate varying levels of visual clutter.

The four 4-dot Braille patterns used as stimuli in both the visual and tactile discrimination tasks.

Computational Architectures

Two ANN architectures were compared to test integration strategies:

  • V-architecture (Late Integration): Two independent networks classify visual and tactile inputs separately. A final layer compares the two labels to determine if they match.
  • Y-architecture (Early Integration): High-level features from both sensory streams are concatenated into a joint "stem" of dense layers before a single output makes the final decision.

Results and Discussion

The Superiority of Early Integration

The Y-architecture consistently outperformed the V-architecture across all stimulus intensities ($p < 0.001$). This suggests that combining sensory information at the feature level allows the system to resolve ambiguities that might lead to errors in a unimodal classification stage.

The "Integration Gap"

When unimodal performance was equalized at the 80% threshold, the following crossmodal accuracies were observed:

  • Younger Humans: 78.3%
  • Y-Architecture (Early Integration): 75.3%
  • V-Architecture (Late Integration): 69.2%
  • Older Humans: 66.2%

The fact that the Y-architecture outperformed older humans (75.3% vs 66.2%) supports the hypothesis that age-related decline is significantly driven by failing integration mechanisms. However, the gap between the best ANN and younger humans (75.3% vs 78.3%) indicates that biological systems possess a superior, yet-to-be-modeled integration efficiency.

Conclusion

This study demonstrates that modern tactile hardware like the BioTac sensor has reached a level of acuity comparable to human fingertips. While early integration architectures provide a significant performance boost for robotic systems, they do not yet fully capture the efficiency of the young human brain. Future research will explore more biologically plausible models, such as Recurrent Neural Networks (RNNs) or Spiking Neural Networks, to better understand the feedback loops and temporal processing that may account for the human advantage.

Summary updated

Abstract

p>The quality of crossmodal perception hinges on two factors: The accuracy of the independent unimodal perception and the ability to integrate information from different sensory systems. In humans, the ability for cognitively demanding crossmodal perception diminishes from young to old age. Here, we propose a new approach to research to which degree the different factors contribute to crossmodal processing and the age-related decline by replicating a medical study on visuo-tactile crossmodal pattern discrimination utilizing state-of-the-art tactile sensing technology and artificial neural networks (ANN). We implemented two ANN models to specifically focus on the relevance of early integration of sensory information during the crossmodal processing stream as a mechanism proposed for efficient processing in the human brain. Applying an adaptive staircase procedure, we approached comparable unimodal classification performance for both modalities in the human participants as well as the ANN. This allowed us to compare crossmodal performance between and within the systems, independent of the underlying unimodal processes. Our data show that unimodal classification accuracies of the tactile sensing technology are comparable to humans. For crossmodal discrimination of the ANN the integration of high-level unimodal features on earlier stages of the crossmodal processing stream shows higher accuracies compared to the late integration of independent unimodal classifications. In comparison to humans, the ANN show higher accuracies than older participants in the unimodal as well as the crossmodal condition, but lower accuracies than younger participants in the crossmodal task. Taken together, we can show that state-of-the-art tactile sensing technology is able to perform a complex tactile recognition task at levels comparable to humans. For crossmodal processing, human inspired early sensory integration seems to improve the performance of artificial neural networks. Still, younger participants seem to employ more efficient crossmodal integration mechanisms than modeled in the proposed ANN. Our work demonstrates how collaborative research in neuroscience and embodied artificial neurocognitive models can help to derive models to inform the design of future neurocomputational architectures.</p>
BibTeX
@article{higgen2020crossmodal,
  author = {Focko L. Higgen and Philipp Ruppel and Michael Görner and Matthias Kerzel and Norman Hendrich and Jan-Frederik Feldheim and Stefan Wermter and Jianwei Zhang and Christian Gerloff},
  title = {{Crossmodal Pattern Discrimination in Humans and Robots: A Visuo-Tactile Case Study}},
  journal = {Frontiers Robotics AI},
  year = {2020},
  doi = {10.3389/frobt.2020.540565},
}
Figure