Abstract
Multimodal Language Models (MLLMs) have shown impressive capabilities in describing static images, but their understanding of dynamic physical environments remains largely unproven. We introduce the GRASP (GRounding And Situated Physics) benchmark to evaluate MLLMs on two critical dimensions: language grounding and intuitive physics. Our findings reveal a significant "perception-reasoning" gap, where state-of-the-art models often perform at chance levels on tasks involving basic physical common sense.
Introduction
While Large Language Models (LLMs) excel at linguistic tasks, they lack "grounding"—the ability to link abstract symbols to physical reality. Multimodal variants (MLLMs) attempt to solve this by incorporating visual data. However, it is unclear whether these models truly understand the physical laws governing the world or are simply identifying statistical patterns in pixels. Existing benchmarks often focus on static scenes or lack a natural language interface. GRASP addresses this by testing models on simulated videos that require both linguistic mapping and physical reasoning over time.
The GRASP Benchmark
The GRASP benchmark utilizes the Unity game engine to generate controlled, high-fidelity video sequences. The dataset comprises 128 videos per test category for Level 1 and 16 distinct physics tests for Level 2, each featuring 128 plausible and 128 implausible video pairs. It is structured into two distinct levels of evaluation:
Level 1: Language Grounding
This level assesses the model's ability to map linguistic descriptors to visual properties. Tasks include identifying object shapes, colors, movement directions, and spatial relationships. For instance, a model might be asked to identify a "red sphere" or determine if an object is "rolling left."
Level 2: Intuitive Physics
Drawing from developmental psychology, Level 2 employs the Violation-of-Expectation (VoE) paradigm. Models are presented with pairs of videos: one "plausible" (physically correct) and one "implausible" (violating laws like gravity, solidity, or continuity). The model must judge the physical possibility of the scene.
Experimental Results and Insights
The researchers evaluated several open-source MLLMs, including Video-ChatGPT, Video-LLaMA, PandaGPT, and VTimeLLM. The results highlight several critical weaknesses in current architectures:
- The Perception-Reasoning Gap: Models that failed basic grounding (Level 1) were almost universally unable to perform physics reasoning (Level 2). Accurate perception is a prerequisite for physical common sense.
- Chance-Level Performance: In zero-shot settings, most models performed near the 50% chance level. While Chain-of-Thought (CoT) prompting improved grounding for static attributes like color, it failed to enhance understanding of motion or physics.
- Human-AI Performance Gap: Human participants achieved an average accuracy of 80% on Level 2 tasks, whereas all tested MLLMs remained at or below 50%.
- Response Bias: Models frequently exhibited a "Yes-Man" bias, agreeing with binary prompts regardless of video content, indicating a lack of genuine analysis.
Conclusion
The GRASP benchmark exposes a fundamental lack of physical understanding in current multimodal AI. The transition from describing images to reasoning about a dynamic world remains a major hurdle. Future research must focus on closing the perception-reasoning gap, potentially through larger model scales or training in interactive, physically-grounded environments.
Summary updated