Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model

Shiori Ueda, Atsushi Hashimoto, Masashi Hamaya, Kazutoshi Tanaka, Hideo Saito

Abstract

Tactile perception is vital, especially when distinguishing visually similar objects. We propose an approach to incorporate tactile data into a Vision-Language Model (VLM) for visuo-tactile zero-shot object recognition. Our approach leverages the zero-shot capability of VLMs to infer tactile properties from the names of tactilely similar objects. The proposed method translates tactile data into a textual description solely by annotating object names for each tactile sequence during training, making it adaptable to various contexts with low training costs. The proposed method was evaluated on the FoodReplica and Cube datasets, demonstrating its effectiveness in recognizing objects that are difficult to distinguish by vision alone.

Introduction

While computer vision has reached incredible heights, vision alone is often insufficient for understanding the world. A plastic orange and a real orange might look identical, and a raw potato looks exactly like a boiled one. Humans use touch to resolve these ambiguities, but integrating tactile data into AI is historically difficult. Standard approaches usually require "fine-tuning" a model on specific sensors and actions, which is expensive and doesn't scale well. If you change the robot's gripper or the type of sensor, you often have to start over.

Conceptual overview: A robot uses tactile sensors to determine if a sushi piece is real or a replica by comparing its feel to known objects like a plastic bottle.

This paper addresses the challenge of zero-shot recognition: identifying objects the robot has never encountered before by leveraging the pre-existing "common sense" knowledge found in Large Vision-Language Models (VLMs) like GPT-4V. By converting tactile signals into text descriptions—specifically by identifying known objects that "feel" similar—the system allows a robot to use its internal knowledge to distinguish between visually identical objects.

Methodology

The researchers developed a pipeline that translates "touch" into "language" without requiring the VLM to understand raw voltage signals from a sensor. This hardware-agnostic approach uses language as a universal interface.

The proposed framework consists of a Tactile Embedding Network for training and a VLM-based inference pipeline that retrieves similar tactile references to generate text prompts.
  1. Tactile Embedding Network: A GRU-based VQ-VAE (Vector Quantized-Variational AutoEncoder) compresses sequences of tactile data (force over time) into discrete digital fingerprints. The use of a Vector Quantization (VQ) layer was found to be crucial for creating a structured tactile space.
  2. Tactile-to-Text Database: During training, the robot touches various reference objects. The system stores these tactile embeddings alongside the object's name.
  3. Inference (The "Bridge"): When encountering a new object, the robot generates a tactile embedding and retrieves the top-k most similar embeddings from its database. It then generates a text prompt: "The mechanical stimulus... is similar to [Object A, Object B, Object C]."
  4. VLM Reasoning: The VLM (e.g., GPT-4V) receives the image and this text description. It uses its internal knowledge (e.g., "plastic bottles are hard, real food is soft") to make a final classification.

Experimental Setup

The framework was evaluated using three distinct datasets:

  • TactileReference: 32 classes (e.g., "banana piece," "sponge") used to build the reference database.
  • FoodReplica: 11 pairs of real food and visually identical wax/plastic replicas, testing material property distinction.
  • Cube: 4 classes of food (squash and rice cakes) processed into uniform cubes, either raw or boiled, testing the detection of internal states that don't change external appearance.

Key Results and Insights

The visuo-tactile method consistently outperformed vision-only baselines:

  • FoodReplica Dataset: The model achieved 58.9% accuracy, compared to 53.6% for vision-only. It was significantly better at identifying replicas, whereas vision-only models tended to label everything as "real food."
  • Cube Dataset: The model reached 65.0% accuracy, a major improvement over the 50.0% (random guessing) of the vision-only baseline.
  • Tactile Common Sense: The most significant finding is that VLMs already "know" how objects should feel. By telling the model "this feels like a plastic bottle," it can deduce the object is likely a replica.

Conclusion

By converting touch to text, the researchers avoided the need to fine-tune the VLM, making the system modular. Future work will explore more diverse gripper motions (sliding, squeezing) to extract richer features and optimize prompt engineering to communicate finer tactile nuances. Currently, "soft touch" approaches designed for fragile items sometimes fail to detect deep internal differences, such as the state of boiled versus raw vegetables.

Summary updated

Abstract

Tactile perception is vital, especially when distinguishing visually similar objects. We propose an approach to incorporate tactile data into a Vision-Language Model (VLM) for visuo-tactile zero-shot object recognition. Our approach leverages the zero-shot capability of VLMs to infer tactile properties from the names of tactilely similar objects. The proposed method translates tactile data into a textual description solely by annotating object names for each tactile sequence during training, making it adaptable to various contexts with low training costs. The proposed method was evaluated on the FoodReplica and Cube datasets, demonstrating its effectiveness in recognizing objects that are difficult to distinguish by vision alone.
BibTeX
@inproceedings{ueda2024visuo,
  author = {Shiori Ueda and Atsushi Hashimoto and Masashi Hamaya and Kazutoshi Tanaka and Hideo Saito},
  title = {{Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801766},
}
Figure