GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models

Serwan Jassim, Mario Holubar, Annika Richter, Cornelius Wolff, Xenia Ohmer, Elia BruniIJCAI 2024

Abstract

Multimodal Language Models (MLLMs) have shown impressive capabilities in describing static images, but their understanding of dynamic physical environments remains largely unproven. We introduce the GRASP (GRounding And Situated Physics) benchmark to evaluate MLLMs on two critical dimensions: language grounding and intuitive physics. Our findings reveal a significant "perception-reasoning" gap, where state-of-the-art models often perform at chance levels on tasks involving basic physical common sense.

Introduction

While Large Language Models (LLMs) excel at linguistic tasks, they lack "grounding"—the ability to link abstract symbols to physical reality. Multimodal variants (MLLMs) attempt to solve this by incorporating visual data. However, it is unclear whether these models truly understand the physical laws governing the world or are simply identifying statistical patterns in pixels. Existing benchmarks often focus on static scenes or lack a natural language interface. GRASP addresses this by testing models on simulated videos that require both linguistic mapping and physical reasoning over time.

The GRASP Benchmark

The GRASP benchmark utilizes the Unity game engine to generate controlled, high-fidelity video sequences. The dataset comprises 128 videos per test category for Level 1 and 16 distinct physics tests for Level 2, each featuring 128 plausible and 128 implausible video pairs. It is structured into two distinct levels of evaluation:

Level 1: Language Grounding

This level assesses the model's ability to map linguistic descriptors to visual properties. Tasks include identifying object shapes, colors, movement directions, and spatial relationships. For instance, a model might be asked to identify a "red sphere" or determine if an object is "rolling left."

Level 2: Intuitive Physics

Drawing from developmental psychology, Level 2 employs the Violation-of-Expectation (VoE) paradigm. Models are presented with pairs of videos: one "plausible" (physically correct) and one "implausible" (violating laws like gravity, solidity, or continuity). The model must judge the physical possibility of the scene.

Overview of GRASP evaluation categories: (a) Shape/Color, (b) Directionality, (c) Object Ordering, and (d) Physics (Solidity).

Experimental Results and Insights

The researchers evaluated several open-source MLLMs, including Video-ChatGPT, Video-LLaMA, PandaGPT, and VTimeLLM. The results highlight several critical weaknesses in current architectures:

  • The Perception-Reasoning Gap: Models that failed basic grounding (Level 1) were almost universally unable to perform physics reasoning (Level 2). Accurate perception is a prerequisite for physical common sense.
  • Chance-Level Performance: In zero-shot settings, most models performed near the 50% chance level. While Chain-of-Thought (CoT) prompting improved grounding for static attributes like color, it failed to enhance understanding of motion or physics.
  • Human-AI Performance Gap: Human participants achieved an average accuracy of 80% on Level 2 tasks, whereas all tested MLLMs remained at or below 50%.
  • Response Bias: Models frequently exhibited a "Yes-Man" bias, agreeing with binary prompts regardless of video content, indicating a lack of genuine analysis.

Conclusion

The GRASP benchmark exposes a fundamental lack of physical understanding in current multimodal AI. The transition from describing images to reasoning about a dynamic world remains a major hurdle. Future research must focus on closing the perception-reasoning gap, potentially through larger model scales or training in interactive, physically-grounded environments.

Summary updated

BibTeX
@inproceedings{jassim2024grasp,
  author = {Serwan Jassim and Mario Holubar and Annika Richter and Cornelius Wolff and Xenia Ohmer and Elia Bruni},
  title = {{GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure