A robot looking at a table sees a ceramic mug and a plastic replica. To a standard computer vision model, these objects may appear identical in shape, color, and texture. However, the moment the robot attempts to lift them, the "visual mirage" breaks: one requires significant torque to overcome its mass, while the other is light and compliant.
This discrepancy highlights a fundamental challenge in robotics: many critical physical properties—such as mass, friction, stiffness, and viscosity—are visually latent. Because these properties are not directly observable from static pixels, relying solely on vision leads to an "ill-posed" problem where multiple physical realities map to the same visual appearance. To resolve this, a growing body of research advocates for interactive multimodal perception, where agents use touch, sound, and purposeful motion to "interrogate" the environment and uncover its hidden mechanical truths.
The Limits of Passive Vision
Traditional computer vision excels at semantic labeling—identifying that an object is a "chair" or a "fabric." But as Anne C. Kemmeren and colleagues point out in Which objects help me to act effectively? Reasoning about physically-grounded affordances, these labels often ignore the subtle material properties required for actual interaction. A plastic chair and a wooden chair have different load-bearing capacities that vision-language models frequently overlook.
This "visual ambiguity" is a recurring theme across the literature. Leonard Friedman noted as early as 1977 in Robot Learning and Error Correction that perceptual systems struggle with objects that look identical but have opposing intrinsic properties like weight. More recently, Yupeng Liang and Ko Nishino demonstrated in Multimodal Material Segmentation that even advanced RGB models cannot reliably distinguish a puddle of water from a dark shadow without auxiliary data like polarization or near-infrared light.
Resolving Ambiguity through Interaction
If vision provides the hypothesis, physical interaction provides the proof. Researchers are increasingly moving toward "active" perception, where robots perform exploratory behaviors to gather ground-truth data.
In DensePhysNet, Jiajun Wu and Andy Zeng 0001 show that a robot can learn a dense map of mass and friction by sliding and colliding with objects. By observing the resulting motion, the system "back-calculates" the physical constants that governed the interaction. Similarly, Amrita S. Sawhney describes a robot "playing with food"—slicing and poking items—to learn representations of ripeness and hardness that are invisible to a camera (Playing with Food).
This approach transforms the robot into an autonomous experimenter. Iain Haughton and Edgar Sucar developed a system that maps a scene's physical "affordances" (like rigidity and friction) from scratch by physically probing objects and updating a 3D neural field in real-time (Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter).
The Sound and Feel of Physics
Beyond motion, two modalities have emerged as primary "truth-tellers" for material properties: haptics and acoustics.
- Haptics (Touch): High-resolution tactile sensors, such as GelSight, allow robots to "see" the deformation of a surface. Haitian Zheng and Lu Fang combined visual textures with vibration signals to achieve near-perfect material classification (Deep Learning for Surface Material Classification Using Haptic And Visual Information). For aerial robots, Xiaofeng Guo and Sebastian A. Scherer integrated these sensors onto UAVs to detect wall defects that were visually indistinguishable from surface patterns (Aerial Interaction with Tactile Sensing).
- Acoustics (Sound): The sound an object makes when struck reveals its internal structure. Auston Sterling and Ming C. Lin developed the Impact Sound Neural Network (ISNN), which identifies material and geometry by "listening" to impacts, a method that succeeds even for transparent or reflective objects where vision fails (ISNN: Impact Sound Neural Network for Audio-Visual Object Classification). Luca Macesanu and Roberto Martín-Martín furthered this with CAVER, a robot that uses curiosity-driven exploration to learn the association between an object's look and its "impact signature" (CAVER: Curious Audiovisual Exploring Robot).
Scaling to Physical Commonsense
The ultimate goal of this research is to move beyond specific classification tasks toward a generalized "physical commonsense." This involves bridging the gap between raw sensory data and high-level reasoning.
Recent work has begun integrating these multimodal signals into Large Language Models (LLMs). Zexiang Guo and Qiang Li developed a framework that feeds visual and tactile sequences into a Vicuna-7B model, allowing it to reason about properties like "squishiness" in natural language (Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference). Similarly, Yifan Xie introduced VTV-LLM, which treats tactile sensor data as "visuo-tactile video," enabling embodied agents to understand the physical consequences of their actions (Universal Visuo-Tactile Video Understanding for Embodied Interaction).
By grounding vision in the "mechanical ground truth" of touch and sound, these systems are beginning to develop an intuitive physics—a "gut feeling" for how the world will react when pushed, pulled, or lifted. As Samuel Yu and Paul Pu Liang argue in their work on the PACS dataset, this multimodal integration is the key to closing the 25% performance gap between AI and human-level physical reasoning (PACS: A Dataset for Physical Audiovisual CommonSense Reasoning).