Abstract
Tactile perception is vital, especially when distinguishing visually similar objects. We propose an approach to incorporate tactile data into a Vision-Language Model (VLM) for visuo-tactile zero-shot object recognition. Our approach leverages the zero-shot capability of VLMs to infer tactile properties from the names of tactilely similar objects. The proposed method translates tactile data into a textual description solely by annotating object names for each tactile sequence during training, making it adaptable to various contexts with low training costs. The proposed method was evaluated on the FoodReplica and Cube datasets, demonstrating its effectiveness in recognizing objects that are difficult to distinguish by vision alone.
Introduction
While computer vision has reached incredible heights, vision alone is often insufficient for understanding the world. A plastic orange and a real orange might look identical, and a raw potato looks exactly like a boiled one. Humans use touch to resolve these ambiguities, but integrating tactile data into AI is historically difficult. Standard approaches usually require "fine-tuning" a model on specific sensors and actions, which is expensive and doesn't scale well. If you change the robot's gripper or the type of sensor, you often have to start over.
This paper addresses the challenge of zero-shot recognition: identifying objects the robot has never encountered before by leveraging the pre-existing "common sense" knowledge found in Large Vision-Language Models (VLMs) like GPT-4V. By converting tactile signals into text descriptions—specifically by identifying known objects that "feel" similar—the system allows a robot to use its internal knowledge to distinguish between visually identical objects.
Methodology
The researchers developed a pipeline that translates "touch" into "language" without requiring the VLM to understand raw voltage signals from a sensor. This hardware-agnostic approach uses language as a universal interface.
- Tactile Embedding Network: A GRU-based VQ-VAE (Vector Quantized-Variational AutoEncoder) compresses sequences of tactile data (force over time) into discrete digital fingerprints. The use of a Vector Quantization (VQ) layer was found to be crucial for creating a structured tactile space.
- Tactile-to-Text Database: During training, the robot touches various reference objects. The system stores these tactile embeddings alongside the object's name.
- Inference (The "Bridge"): When encountering a new object, the robot generates a tactile embedding and retrieves the top-k most similar embeddings from its database. It then generates a text prompt: "The mechanical stimulus... is similar to [Object A, Object B, Object C]."
- VLM Reasoning: The VLM (e.g., GPT-4V) receives the image and this text description. It uses its internal knowledge (e.g., "plastic bottles are hard, real food is soft") to make a final classification.
Experimental Setup
The framework was evaluated using three distinct datasets:
- TactileReference: 32 classes (e.g., "banana piece," "sponge") used to build the reference database.
- FoodReplica: 11 pairs of real food and visually identical wax/plastic replicas, testing material property distinction.
- Cube: 4 classes of food (squash and rice cakes) processed into uniform cubes, either raw or boiled, testing the detection of internal states that don't change external appearance.
Key Results and Insights
The visuo-tactile method consistently outperformed vision-only baselines:
- FoodReplica Dataset: The model achieved 58.9% accuracy, compared to 53.6% for vision-only. It was significantly better at identifying replicas, whereas vision-only models tended to label everything as "real food."
- Cube Dataset: The model reached 65.0% accuracy, a major improvement over the 50.0% (random guessing) of the vision-only baseline.
- Tactile Common Sense: The most significant finding is that VLMs already "know" how objects should feel. By telling the model "this feels like a plastic bottle," it can deduce the object is likely a replica.
Conclusion
By converting touch to text, the researchers avoided the need to fine-tune the VLM, making the system modular. Future work will explore more diverse gripper motions (sliding, squeezing) to extract richer features and optimize prompt engineering to communicate finer tactile nuances. Currently, "soft touch" approaches designed for fragile items sometimes fail to detect deep internal differences, such as the state of boiled versus raw vegetables.
Summary updated