A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval

Jagoda Wojcik, Jiaqi Jiang, Jiacheng Wu, Shan Luo

Introduction

In robotics, Cross-Modal Retrieval (CMR) is the ability to find information in one sensory format (like audio) based on a query from another (like a picture). While humans do this effortlessly—hearing a "clink" and knowing it’s a metal spoon without looking—robots struggle. Most current robotic CMR systems are bi-modal, meaning they only link two senses (e.g., vision and touch). When you add a third sense, like audio, the complexity spikes. Simply "glueing" (concatenating) these different data types together often results in noisy, muddy representations that degrade performance.

Overview of the Cross-Modal Retrieval training and testing pipeline.

This paper introduces VAT-CMR, a framework designed to create a unified "shared space" where visual, audio, and tactile data can be accurately aligned and retrieved.

Methodology

The authors propose the VAT-CMR (Visual-Audio-Tactile Cross-Modal Retrieval) framework, which uses a three-branch neural network architecture to process disparate sensory inputs.

The VAT-CMR architecture, illustrating feature extraction, Multi-Head Attention fusion, and the Triplet Loss mechanism.

Feature Extraction

  1. Vision & Tactile: Two separate ResNet-50 models (pre-trained and fine-tuned) extract features from RGB images and optical tactile sensor images (like GelSight).
  2. Audio: A 1D-CNN processes audio spectrum patches to create embeddings.

Multi-Head Attention (MHA) Fusion

Instead of simple averaging or concatenation, the model uses MHA to highlight shared features between modalities. This preserves the "integrity" of the features, ensuring that critical information from one sense isn't washed out by another.

Dominant Modality Selection

This is the paper's "secret sauce." During the classification training phase, instead of optimizing based on a combined vector of all senses, the researchers identify and focus on the most "informative" modality (the dominant one) to guide the learning process. This reduces complexity and helps the model learn more distinct, meaningful representations.

Common Latent Space

Finally, a triplet loss function is used to pull representations of the same object closer together in a shared mathematical space while pushing different objects further apart.

Experimental Setup

The study utilizes a synthetic dataset derived from ObjectFolder 2.0, consisting of 34,500 samples across 20 different objects (ranging from dustpans and bells to toy bears). Each object has paired RGB images (vision), audio spectrograms (impact sounds), and tactile images (simulated optical touch). The data is split into 25,500 samples for training, with the remainder used for validation and testing.

Key Results

The VAT-CMR model was compared against standard baselines like Canonical Correlation Analysis (CCA) and the original ObjectFolder retrieval method.

  • Superior Accuracy: VAT-CMR achieved a Mean Average Precision (MAP) of 0.96 for "Touch + Audio" retrieval given a visual query, a massive improvement over the ObjectFolder baseline of 0.67.
  • Cross-Modal Versatility:
    • Vision → Touch: 0.93 MAP (VAT-CMR) vs 0.51 (Baseline).
  • Audio → Vision: 0.77 MAP (VAT-CMR) vs 0.52 (Baseline).
  • Ablation Success: Removing the dominant modality selection caused an average MAP reduction of 0.17, confirming it as a critical component of the architecture.

Discussion and Future Work

The experiments showed a "fixed relationship" between certain modalities. For instance, when audio is involved, selecting it as the dominant modality during training led to the highest retrieval scores (up to 0.94 MAP). This suggests that focusing on a single "dominant" modality during training acts as a stable anchor for the other modalities to align with.

While the results are impressive, the authors note several areas for growth, specifically the Sim2Real Gap. The current study relies entirely on synthetic data, whereas real-world physics and sensor noise are much more unpredictable. Future work will explore active exploration, where a robot might choose to poke or move an object specifically to gather more informative audio or tactile data, and the incorporation of additional modalities like force feedback.

Summary updated

BibTeX
@inproceedings{wojcik2024case,
  author = {Jagoda Wojcik and Jiaqi Jiang and Jiacheng Wu and Shan Luo},
  title = {{A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802105},
}
Figure