Guest editorial

Huaping Liu, Yuan YuanIndustrial 2019 · DOI · OpenAlex

Introduction

In the realm of industrial robotics, the ability to perceive the environment through multiple sensory channels is a critical frontier. While humans naturally combine sight, sound, and touch, robots typically struggle to integrate disparate data sources. This special issue of Industrial Robot explores cross-modal perception—the process of learning knowledge from one modality (such as vision) to enhance understanding in another (such as touch). The goal is to establish a unified representation that allows robots to navigate and interact with their surroundings more effectively.

The Challenges of Multi-Modal Integration

Despite the proliferation of sensors in modern robotics, two primary challenges hinder effective cross-modal learning:

  1. The Heterogeneous Gap: Different sensors produce data in vastly different feature spaces. For instance, comparing the RGB values of a camera to the pressure readings of a tactile sensor is inherently difficult due to their distinct mathematical structures.
  2. Weak Pairing: It is often difficult to establish a direct correspondence between samples from different modalities. Identifying exactly which visual feature correlates with a specific tactile sensation remains a significant technical hurdle.

Research Pillars: Sensing, Perception, and Applications

The 15 papers selected for this special issue address these challenges across three key areas:

1. Sensing

Innovation at the hardware level is fundamental to cross-modal perception. Research in this category includes:

  • Tactile Force Measurement: New sensor designs specifically for measuring robotic grasping forces.
  • Bio-Inspired Models: Tactile texture sensors based on the inverse-magnetostrictive effect, mimicking biological perception of stimuli-current thresholds.
  • Artificial Skin: Integrating skin-like signals into reactive control systems for safer human-robot collaboration.

2. Perception

Bridging the gap between sensors requires advanced algorithmic frameworks. Featured techniques include:

  • Image Fusion: Utilizing guided filters and redundant discrete wavelet transforms to merge visual data.
  • Similarity Learning: Using weakly-paired similarity learning to help robots identify surface materials by correlating visual and tactile data.
  • Transfer Learning: Applying knowledge across domains, such as road segmentation in remote sensing.

3. Applications

The ultimate test of cross-modal perception is its performance in real-world scenarios:

  • Autonomous Systems: Combining radar and vision for improved obstacle detection in autonomous vehicles.
  • Complex Control: Using Reinforcement Learning (RL) for satellite attitude adjustment and exoskeleton stair-ascent planning.
  • Industrial Precision: Developing specialized methods for welding seam identification.

Key Insights and Future Directions

The research highlights that unified representation is the cornerstone of cross-modal perception. Rather than simply overlaying data, the most effective systems use one modality to "teach" another, creating a more robust understanding of the environment.

However, limitations remain. Most current solutions are task-specific, and the field still lacks a general-purpose framework that can scale across diverse industrial applications. Future research must continue to address the "heterogeneous gap" and find ways to overcome the scarcity of strongly paired training data.

Source: Industrial Robot: the international journal of robotics research and application, Vol. 46 No. 3, 2019. DOI: 10.1108/IR-05-2019-458

Summary updated

Abstract

Special issue on cross-modal perception for industrial robotsCross-modal perception means learning knowledge from some modality (e.g.vision) and using this knowledge to understand the environment with another modality (e.g.touch).The key to cross-modal perception is to establish a unified representation between multi-modal data sources.Cross-modal learning has been extensively used in Web search and human-computer interaction, but it finds little application in industrial robots.Currently, we can find several sensors in industrial robots, but the multi-modal integration and cross-modal transferring problems are still difficult.The first challenge stems from the heterogeneous gap: there exists significant feature space difference between modalities, and direct comparison or matching between them is difficult.The second challenge stems from the weak pairing: it is difficult to establish the sample correspondence relationship between modalities.These challenges incur many unsolved problems worthy of research.This special issue mainly focuses on cross-modal perception and learning technology for modern industrial robots, solving both original theoretical development and engineering applications of cross-modal learning and transferring.We received many high-quality submissions, and only 15 have been finally accepted for publication after several rounds of peer review.They can be divided into three categories: sensing, perception and applications.A few papers solve some basic sensing problems for crossmodal perception tasks.In "A cross-modal tactile sensor design for measuring robotic grasping forces," a cross-modal tactile sensor design for measuring robotic grasping forces has been developed.In "The perceptually-inspired model of tactile texture sensor based on the inverse-magnetostrictive effect) has described the development of a new perceptually inspired model of tactile texture sensor based on the inversemagnetostrictive effect.The paper "An investigation of stimuli-current thresholds on the non-steady contact condition" describes the investigation of the stimuli-current thresholds on the non-steady contact condition.In addition, the authors of "Incorporating artificial skin signals in the constraint-based reactive control of human-robot collaborative manipulation tasks" incorporate artificial skin signals in the constraint-based reactive control of humanrobot collaborative manipulation tasks.All of these works
BibTeX
@inproceedings{liu2019guest,
  author = {Huaping Liu and Yuan Yuan},
  title = {{Guest editorial}},
  booktitle = {Industrial},
  year = {2019},
  doi = {10.1108/ir-05-2019-458},
}

Loading related content...

Figure