Can Your Model Separate Yolks with a Water Bottle? Benchmarking Physical Commonsense Understanding in Video Generation Models

Baris Sarper Tezcan, Enes Sanli, Erkut Erdem, Aykut ErdemSubmission to ICLR 2026 · Rejected

TL;DR

While modern text-to-video (T2V) models generate visually stunning scenes, they consistently fail at everyday physical commonsense—especially tool use, affordances, and causal interactions. PhysVidBench introduces a human-validated benchmark of 383 everyday physical scenarios (766 base and upsampled prompts) paired with a modular evaluation pipeline that decouples video perception from reasoning. The evaluation shows that even cutting-edge models like Sora, Veo-2, and Cosmos-14B rarely score above 36% accuracy on physical commonsense tests.

What Problem Does This Paper Address?

State-of-the-art video generators are frequently pitched as prospective "world models" for embodied AI and robotics. However, behind photorealistic textures lies a fragile grasp of physics: tools clip through surfaces, liquids disobey gravity, and multi-step causal chains collapse.

Existing video physics benchmarks primarily probe isolated textbook phenomena (like buoyancy or simple falling balls) or rely on direct Vision-Language Model (VLM) scoring. Direct VLM scoring often fails due to severe hallucinations and "mode collapse" (e.g., answering "Yes" to every query regardless of video content). PhysVidBench targets everyday physical commonsense—specifically purposeful tool manipulation, object affordances (such as repurposing a plastic bottle as an egg separator), and multi-step procedural causality.

Methods

PhysVidBench tackles benchmark creation and evaluation via two key designs:

  1. Task & Prompt Design (Derived from PIQA):
  2. Scenarios are selected from the physical QA dataset PIQA, explicitly filtered for secondary tool use and non-obvious affordances (e.g., using a credit card as an ice scraper).
  3. Prompts are generated in two tiers: a concise base prompt and an upsampled prompt that enriches the scene with explicit object attributes, force application, and material constraints without changing the core action.
  4. Modular Caption-Based Evaluation (Decoupling Perception from Reasoning):
  5. Physics-Grounded Questions: Gemini 2.5 Pro generates targeted Yes/No questions (averaging 11 per prompt) where the expected answer is always "Yes" if the physics occurred correctly.
  6. Dense, Multi-Perspective Captioning: Instead of asking a VLM to judge the video directly, AuroraCap generates 8 separate captions per video: one general caption plus seven dimension-specific captions focused on forces, spatial layouts, material transformations, etc.
  7. LLM-as-a-Judge: An LLM (Gemini 2.5 Flash) answers the physics questions strictly using evidence present in the 8 dense captions.
  8. Training-Free Prompt Refinement: The benchmark can act as a diagnostic feedback loop: detected physical violations are fed into an LLM to iteratively rewrite generation prompts, improving physical realism without model retraining.
[Prompt] ──> [T2V Model] ──> [Video] ──> [AuroraCap (8 Dense Captions)]
                                                      │
[Upsampled Prompt] ──> [Targeted Yes/No Questions] ───┴──> [LLM Judge] ──> [Score]

Datasets

  • PIQA: Source dataset for everyday physical goals and solutions.
  • PhysVidBench (New Benchmark):
  • 383 base prompts and 383 upsampled variants (766 prompts total).
  • 4,123 manually audited, physics-grounded Yes/No questions across 7 ontology dimensions: Fundamental Physics, Object Properties & Affordances, Spatial Reasoning, Temporal Dynamics, Action & Procedural Understanding, Material Interaction & Transformation, and Force & Motion.
  • Stratified into three difficulty tiers: Medium (123 prompts), Hard (160 prompts), and Very Hard (100 prompts).

Key Insights and Ideas

  • Decoupled Evaluation Crushes Direct VLM Scoring: Direct video-QA with VLMs (e.g., Video-LLaVA, InternVL3) suffers from mode collapse, often predicting "Yes" across the board. The caption-then-reason approach achieves high Pearson correlation with human raters ($r = 0.45\text{–}0.69$), whereas standard video metrics (like VideoScore) yielded near-zero or negative correlations.
  • Relational and Temporal Physics are the Hardest: Across all models, Spatial Reasoning (geometry, fit, occlusion) and Temporal Dynamics (delays, ordering) consistently yielded the lowest scores ($13\%\text{–}29\%$). Models perform highest on Object Properties & Affordances ($25\%\text{–}41\%$), largely by leaning on surface visual heuristics (textures and silhouettes) rather than causal mechanics.
  • Model Scale Helps, But Quickly Plateaus: Wan2.1-14B beats Wan2.1-1.3B, but CogVideoX-2B outperforms CogVideoX-5B. Architecture, motion priors, and pre-training data often matter more than raw parameter count.
  • Prompt Upsampling Unlocks Latent World Knowledge: Providing explicit physical descriptions significantly improves open-source models (e.g., Cosmos-14B gains +7.1 points). Proprietary models like Sora slightly degraded, likely due to interference with proprietary internal prompt-expansion modules.

Key Results

  • Overall Rankings: The top-performing models were Cosmos-14B (36.2%), Veo-2 (34.8%), and Wan2.1-14B (34.1%), followed by MAGI-1 (32.6%) and Sora (31.4%). VideoCrafter2 (22.1%) and LTX-Video (21.0%) lagged behind.
  • Steep Difficulty Dropoff: On upsampled prompts, average scores drop precipitously from the Medium subset (up to 61.5% for Wan2.1-14B) to the Very Hard subset (between 9.1% and 22.2% across all models).
  • Prompt Refinement Gains: Using PhysVidBench as a diagnostic loop improved CogVideoX-2B from 21.6% to 32.7% (+29.8% relative gain) and CogVideoX-5B from 17.8% to 29.7% (+38.1% relative gain) on a 100-prompt subset without retraining.

Limitations and Future Work

  • Reliance on Caption Quality: The LLM judge never sees the raw video pixels. While cross-captioner experiments (swapping to Qwen-2.5-VL / Qwen-3-VL) demonstrated rank consistency ($\rho = 0.80\text{–}0.98$), subtle visual physics that evade text description cannot be scored.
  • Proprietary Model Access: The benchmark primarily targets open-source models for reproducibility, evaluating only limited runs on closed systems like Sora and Veo-2.
  • Held-Out Subset: The 100 "Very Hard" prompts are currently reserved by the authors for future evaluations, with only Medium and Hard sets publicly released.

Summary updated

Abstract

Recent advances in text-to-video (T2V) generation have enabled visually compelling outputs, but models still struggle with everyday physical commonsense, often producing videos that violate intuitive expectations of causality, object behavior, and tool use. We introduce PhysVidBench, a human-validated benchmark for assessing physical reasoning in T2V models. It comprises carefully curated prompts spanning seven dimensions of physical interaction, from material transformation to temporal dynamics, offering broad, multi-faceted coverage of scenarios where physical plausibility is critical. For each prompt, we generate videos using diverse state-of-the-art models, and evaluate them through a three-stage pipeline: grounded physics questions are derived from each prompt, generated videos are captioned with a vision–language model, and a language model answers the questions using only the captions. This strategy mitigates hallucination and produces scores that align closely with human judgments. Beyond evaluation, PhysVidBench also serves as a diagnostic tool, enabling feedback-driven refinement of model outputs. By emphasizing affordances and tool-mediated actions, areas often overlooked in existing benchmarks, PhysVidBench provides a structured, interpretable framework for assessing and improving everyday physical commonsense in T2V models.
BibTeX
@misc{tezcan2025can,
  author = {Baris Sarper Tezcan and Enes Sanli and Erkut Erdem and Aykut Erdem},
  title = {{Can Your Model Separate Yolks with a Water Bottle? Benchmarking Physical Commonsense Understanding in Video Generation Models}},
  year = {2025},
  eprint = {2507.15824},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.15824},
}
Figure