TL;DR
While modern Video Large Language Models (Video-LLMs) excel at general video understanding, they often act as sophisticated pattern recognizers that fail to grasp underlying physical laws. To bridge this gap, the authors introduce PhyCheck, a new dataset of nearly 70,000 question-answer pairs designed to evaluate and train Video-LLMs on physical causality through multi-grained, evidence-grounded reasoning.
What Problem Does This Paper Address?
For AI to evolve into true "World Models" capable of safely interacting with the physical world (as required in embodied intelligence), they must understand physical regularities like gravity, friction, and conservation laws. However, current Video-LLMs frequently rely on superficial statistical shortcuts rather than a causal understanding of physics. They might recognize a bowling ball hitting pins, but struggle to reliably determine if the resulting motion violates the conservation of momentum.
Existing benchmarks primarily evaluate the visual quality of generated videos or rely on coarse-grained, isolated common-sense questions. They lack the granularity needed to diagnose why a model thinks a video is physically plausible or implausible. Furthermore, physical validity is often context-dependent (e.g., a floating ball is impossible in isolation, but perfectly reasonable if an active hairdryer is blowing underneath it).
Methods
The authors address these limitations by building a hierarchical, multi-tier Video Question Answering (VQA) framework that decomposes complex physical laws into explicit visual evidence clues.
The dataset construction relies on a multi-stage pipeline:
- Rule-Grounded Physical VQA: The authors leverage synthetic videos from the VideoPhy2 benchmark, mapping them to specific physical rules. They use template-based generation to create balanced "consistency" and "violation" questions to avoid fixed-answer biases.
- Evidence-Grounded VQA: To move beyond simple yes/no judgments, they use Gemini-3 to analyze the videos and generate "negative prompts" (hypotheses of what physical details went wrong). GPT-5 then refines these into fine-grained questions probing explicit visual evidence (e.g., "Did the object move without an external force?").
- Human Verification: Two independent human annotators verify every question-answer pair to ensure visual grounding and accuracy, achieving a remarkably high raw agreement rate of 99.74%.
Additionally, the authors conduct a Context-Sensitive Physical Evaluation pilot study. They manually edit real-world videos to crop out causal factors (like removing the hairdryer under a floating ball) to see if models can dynamically update their physical judgments once the external text context is provided.
Datasets
The primary contribution is the PhyCheck dataset:
- Videos: 6,399 synthetic videos (totaling 8.61 hours) generated by seven state-of-the-art text-to-video models (including Sora, Luma Ray2, and CogVideo).
- QA Pairs: 69,825 binary (yes/no) question-answer pairs, averaging 10.9 questions per video.
- Categories: Questions span six major physical principles: Mechanics Laws, Conservation Laws, Material Properties, Fluids/Interface Phenomena, Optical Laws, and Thermal Laws.
- Context-Sensitive Subset: A specialized, hand-crafted diagnostic subset consisting of 50 video-context pairs.
Key Insights and Ideas
- Superficial Success, Deep Vulnerability: Off-the-shelf Video-LLMs show a strong performance bias toward assuming videos are physically consistent. They score well on standard positive samples but fail sharply when encountering negative, counter-physical anomalies.
- Intuitive vs. Implicit Physics: Models perform significantly better on visually intuitive phenomena (like Inertia, Reflection, and Buoyancy) where cues are directly observable. They struggle heavily with implicit physical reasoning, such as Conservation Laws and Newtonian mechanics, which require inferring latent states.
- The "Fine-to-Coarse" Learning Trajectory: When fine-tuning models, training order matters immensely. Training a model on fine-grained physical details before introducing coarse-grained physical reasoning yields the best results. This suggests that mastering foundational physical concepts acts as a necessary scaffolding for higher-level reasoning.
Key Results
- Baseline Performance: Most advanced open-source and proprietary models (including the Qwen series and Gemini-3.5-Flash) achieve mediocre performance on PhyCheck, hovering around an overall accuracy of 60%.
- Fine-Tuning Impact: By fine-tuning Qwen2.5-VL on PhyCheck using the optimal sequential strategy (fine-grained VQA followed by coarse-grained VQA), overall accuracy surged to 81.00%.
| Training Strategy | Overall Accuracy |
|---|---|
| Baseline (No Fine-Tuning) | 50.93% |
| Fine-grained VQA Only | 51.43% |
| Coarse-grained VQA Only | 66.83% |
| Fine-grained $\rightarrow$ Coarse-grained (O2 $\rightarrow$ O1) | 81.00% |
- Context-Sensitivity Deficit: In the pilot study, baseline models achieved high accuracy in the video-only setting by prematurely flagging partial observations as physical violations. However, they failed to update their predictions when given the missing causal context. In contrast, the fine-tuned model successfully adapted, with its context-assisted performance rising to 0.98.
Limitations and Future Work
The authors note that their context-sensitive diagnostic subset is currently limited to a small scale (50 pairs) due to the intensive manual video editing required. While fine-tuning on PhyCheck significantly mitigates failure rates across all physical categories, current models still struggle to seamlessly integrate external causal conditions into their broader reasoning chains. Future work will need to scale up context-aware physical evaluations to help Video-LLMs completely move past rigid visual shortcuts.
Summary updated