Abstract
Recent advancements in text-to-video and text-to-image models have led to their characterization as potential "world simulators." However, existing evaluation metrics primarily focus on perceptual fidelity and semantic alignment, often overlooking the underlying physical and logical rules governing the real world. This paper introduces MMGR (Multi-Modal Generative Reasoning), a comprehensive benchmark designed to evaluate generative models across five reasoning pillars: Physical, Logical, 3D Spatial, 2D Spatial, and Temporal. Through an evaluation of 1,853 samples across abstract reasoning, embodied navigation, and physical commonsense, we reveal a stark "illusion of competence." Our findings show that while models can generate visually stunning content, they often fail to maintain causal and logical consistency, with video models frequently underperforming compared to their image-based counterparts on reasoning tasks.
Introduction
The generative AI landscape has transitioned from static image generation to complex video synthesis with models like Sora, Veo, and Midjourney. As these models produce increasingly photorealistic visuals, the research community has begun to question whether they truly internalize the laws of physics and logic. Current standard metrics, such as Fréchet Video Distance (FVD) and CLIP similarity, are insufficient for this purpose; they measure how "real" a frame looks or how well it matches a text prompt, but they remain blind to whether a billiards ball passes through another or if a robot teleports through a wall.
To act as true world simulators, models must move beyond pixel-perfect reconstruction and demonstrate an understanding of spatial geometry, fluid intelligence, and physical constraints. MMGR addresses this gap by providing a rigorous, reasoning-focused evaluation framework that moves the goalpost from visual plausibility to structural correctness.
The MMGR Benchmark
Evaluation Framework
MMGR evaluates models across three primary domains, utilizing a Vision-Language Model (VLM) judge—specifically Gemini 2.5-Pro—to perform fine-grained analysis. To ensure the highest standards, the benchmark employs a "Holistic Overall" metric, where a generation is only considered successful if it satisfies every sub-metric (e.g., maintaining layout consistency while solving a maze).
- Abstract Reasoning: This domain tests fluid intelligence through programmatically generated Mazes, Sudoku grids, and mathematical challenges (GSM8K, MATH500, Omni-MATH). It also includes the ARC-AGI benchmark to test non-verbal reasoning.
- Embodied Navigation: Models are tasked with navigating 3D environments sourced from Matterport3D and HM3D. Tasks include generating panoramic views, top-down maps, and Simultaneous Localization and Generation (SLAG).
- Physical Commonsense: This evaluates the generation of fundamental physics (fluid dynamics, rigid body collisions) and complex compositional sports scenarios like ballet or skiing.
Dataset Composition
The benchmark comprises 1,853 testing samples. A critical component of the methodology is the validation of the automated VLM judge against human evaluation, which helps identify "physics violations" that automated systems might otherwise overlook.
Key Insights and Results
The Illusion of Competence
A primary finding of the MMGR benchmark is that visual realism is entirely decoupled from physical plausibility. For instance, the model Wan-2.2 produces videos with high visual realism scores (84-96%) but fails significantly on physical grounds, achieving only a 24% overall success rate. Models often mimic the appearance of a process—such as a ball rolling toward a goal—while hallucinating the intermediate steps or violating basic spatial logic.
The "Temporal Tax" on Reasoning
One of the more counterintuitive findings is that video models perform significantly worse than image models on logical tasks. The computational and architectural overhead required to maintain frame-to-frame visual coherence appears to compete with the model's ability to maintain logical consistency. In Sudoku tasks, for example, Veo-3 achieved a 0% overall success rate under human evaluation, whereas image-centric models like Nano-banana Pro dominated abstract reasoning tasks, achieving 97.8% on GSM8K math problems.
Memorization vs. Generalization
The benchmark highlights a heavy reliance on training data patterns rather than true reasoning. Sora-2 showed decent performance (~20%) on the older ARC-AGI v1 dataset but collapsed to a 1.3% success rate on the newer v2 dataset. This suggests that current generative models are largely interpolating within their training distribution rather than developing the ability to solve novel, out-of-distribution logical puzzles.
Navigation and Spatial Awareness
While models like Veo-3 show promise in simple ego-centric navigation (60% success in Panoramic Last-Mile tasks), they fail in complex scenarios requiring cross-view alignment. In the SLAG task, which requires consistent localization and generation, success rates plummeted to 3.6%, indicating a lack of persistent 3D spatial memory.
Discussion and Future Work
The failures identified in MMGR point toward three fundamental bottlenecks in the current generation of "world simulators":
- Data Imbalance: Training corpora are saturated with naturalistic video but lack structured, symbolic reasoning data.
- Architectural Limitations: Current models lack a persistent global state or "memory" of the world, leading to transient physics violations.
- Objective Mismatch: Optimization for pixel-level reconstruction rewards visual plausibility over causal correctness.
Future research must focus on architectures that decouple internal reasoning states from visual rendering. Integrating neuro-symbolic supervision or reinforcement learning from structured physical feedback may be necessary to move beyond the illusion of competence and toward genuine multi-modal reasoning.