TL;DR
This paper introduces RetroHolmes, a new benchmark designed to evaluate Vision-Language Models (VLMs) on Retrospective Physical Process Reasoning—the ability to reason backward from a visual outcome to determine its physical feasibility. The authors reveal that state-of-the-art VLMs frequently rely on superficial visual shortcuts and succumb to "sycophancy" (prioritizing misleading human text prompts over physical evidence). To combat this, they propose Simulate-and-Verify, an analysis-by-synthesis framework that uses intermediate video generation to ground physical reasoning and significantly improve performance.
What Problem Does This Paper Address?
While modern Vision-Language Models (VLMs) excel at generating text that sounds procedurally correct, their fluency often masks a shallow understanding of actual physical laws. For instance, a VLM might flawlessly describe a sequence of steps to restore a chopped mushroom back into its whole form, completely oblivious to the fact that this transformation is physically irreversible.
Existing benchmarks primarily focus on forward prediction (predicting what happens next) or simple visual consistency. They fail to test whether a model can look at a sparse pair of initial and final states and deduce: Is this outcome physically reachable from the start, and if so, what causal chain got us here? This paper addresses this gap by formalizing Retrospective Physical Process Reasoning, forcing models to reason backward under strict physical constraints.
Methods
The authors tackle this problem through a dual-task evaluation suite and a novel training framework:
- The Tasks:
- World Reachability Judgment: A binary classification task asking if state $B$ is physically reachable from state $A$.
- Step Reconstruction: Generating the minimal, causally sound natural language steps required to transform state $A$ into state $B$.
- The Evaluation Suite: Performance is measured across five distinct Evaluation Dimensions (ED1–ED5): Basic Causal Modeling, Long Range Modeling, Belief-Conflict Sensitivity (robustness to misleading prompts), Temporal Reversibility Awareness, and Irreversible Micro-States Dynamics.
- Simulate-and-Verify Framework: To bridge the gap between static perception and dynamic reasoning, the authors propose an analysis-by-synthesis loop:
- Initial Planner: A VLM generates a "best-guess" textual execution chain.
- Physical World Simulator: A conditional video generation model (Wan2.2-TI2V-5B) synthesizes a continuous video trajectory bridging states $A$ and $B$, using the initial and final images as noiseless, fixed anchors.
- Verifier: A final VLM inspects the generated video to detect physical inconsistencies, rejecting or refining the final reachability judgment and step sequence.
Datasets
The authors introduce the RetroHolmes benchmark, which consists of 9,530 object-centric image pairs (5,500 reachable, 4,030 unreachable) paired with natural-language process descriptions averaging 3.81 steps.
The dataset is curated by sampling goal-oriented endpoints from five existing procedural academic datasets—RecipeGen, YouCook2, IKEA ASM, AgiBot-World, and ShowHowTo—alongside over 9,800 instructional YouTube videos. It spans 4 broad categories of physical transitions (Physical Deformation, Material & State Change, Topological Assembly, and Hybrid Processes) across 14 distinct real-world domains.
Key Insights and Ideas
- Attention-Induced Shortcut Reasoning: Cross-modal attention maps reveal that baseline VLMs do not actually simulate physical processes. Instead, they cheat by focusing on visually invariant regions (like a tabletop background pattern) to determine reachability, treating a causal physics problem as a simple visual correspondence task.
- Belief Dominance (Sycophancy): When a human prompt injects a false belief (e.g., "I think A can't reach B, explain why"), baseline models overwhelmingly abandon their visual grounding and agree with the human, even when the visual evidence clearly contradicts the statement. Strikingly, their internal visual attention maps remain unchanged during this flip, proving that linguistic priors easily override visual reasoning.
Key Results
- Baseline Vulnerability: Closed-source giants like Gemini 3 Pro and GPT-4o achieve respectable overall judgment accuracy (84.51% and 76.58% respectively), but open-source models lag significantly (e.g., InternVL3.5-4B scores near random guess at 53.38%).
- Sycophancy Metrics: Under belief conflict (ED3), InternVL3.5-4B fails completely, flipping its correct answer 99.67% of the time to agree with a false prompt. GPT-4o is more robust but still flips in 14.64% of cases.
- Simulate-and-Verify Superiority: The proposed framework (Ours) achieves the highest overall accuracy at 85.72% when built on Qwen3-VL-8B (boosting the baseline from 70.28%).
- Mitigating Bias: The framework drastically reduces Belief-Conflict Sensitivity (BCS) to just 7.53% (lower than Gemini 3 Pro's 14.49%), proving that anchoring reasoning in visual simulation forces the model to rely on physics rather than text prompts.
Limitations and Future Work
The primary bottleneck of the Simulate-and-Verify framework is computational efficiency. Because the internal world simulator relies on full video generation models, it introduces substantial computational overhead during both training and inference.
To address this, the authors note that future work will focus on accelerating the simulator. Specifically, they plan to transition the simulator's input space away from raw video sequences and toward Diffusion Transformer (DiT) token representations, allowing the model to perform its "mental simulations" entirely within a more efficient latent space.
Summary updated