Abstract
Counterfactual reasoning, a crucial manifestation of human intelligence, involves making presuppositions based on established facts and extrapolating potential outcomes. While existing multimodal large language models (MLLMs) exhibit impressive cognitive capabilities on standard benchmarks, their performance on counterfactual questions remains under-explored. We introduce the CounterFactual MultiModal (CFMM) reasoning benchmark (also known as MARS-Bench) to systematically assess these capabilities across six challenging tasks. Our findings reveal that existing MLLMs often "believe what they see," ignoring counterfactual presuppositions in favor of empirical intuition. To address this, we evaluate a wide range of MLLMs and propose potential avenues for improvement toward human-level intelligence.
Introduction: The Intuition Trap
Current MLLMs like GPT-4o and Qwen2.5-VL often rely on "empirical intuition"—statistical patterns from their training data—rather than actual logical deduction. This makes them susceptible to being fooled by hypothetical "what if" questions. For example, if asked, "If I painted this bus blue, what color would it be?" many models default to the original color seen in the image. This over-reliance on pattern matching prevents models from performing true compositional reasoning, which requires breaking a problem into logical, hypothetical steps.
The MARS-Bench (CFMM) Dataset
To systematically evaluate these failures, we developed the Multimodal Assumptive Reasoning Benchmark (MARS-Bench). The dataset consists of 1,022 images and 6,000 questions categorized into six dimensions: Count, Color, Size, Shape, Direction, and Common Sense. Each image is paired with a fact-based "basic" question and several "assumptive" (counterfactual) questions that require the model to mentally simulate a change to the scene.
Methods: Active Deduction (AD)
The researchers proposed a two-stage training paradigm called Active Deduction (AD) to move models away from simple pattern matching:
- AD-SFT (Supervised Fine-Tuning): The model is trained on a curated dataset where complex questions are paired with structured reasoning paths wrapped in
<think>tags. Crucially, simple questions are answered directly, teaching the model to only "think" when necessary. - AD-RFT (Reinforcement Fine-Tuning): Using Group Relative Policy Optimization (GRPO), the model is refined based on format rewards (correct use of reasoning tags) and accuracy rewards (evaluated by a judge model like GPT-4o-mini).
This "Active" approach allows the model to dynamically adjust its cognitive effort based on the difficulty of the prompt, avoiding the unnecessary computational cost of forced reasoning on simple tasks.
Key Results
Experiments demonstrate that the Active Deduction framework significantly closes the reasoning gap for open-source models:
- Massive Accuracy Gains: On MARS-Bench assumptive questions, the Qwen-AD 7B model achieved an accuracy score of 495.2, a staggering 76.6-point improvement over the baseline Qwen2.5-VL 7B (418.6).
- No "Reasoning Tax": Unlike models forced to "think" for every prompt, Qwen-AD maintained or improved its performance on standard, non-assumptive benchmarks like OCRBench and LLaVABench.
- Deduction over Intuition: By training the model to decompose problems (e.g., identifying a direction before simulating a U-turn), the model moves toward human-like deduction.
Conclusion and Future Work
While the Qwen-AD series significantly improves robustness, challenges remain. The current reward system relies heavily on the biases of judge models like GPT-4o-mini, and extremely complex, multi-layered logic scenarios still pose a challenge for 7B-parameter models. Future research will focus on expanding data diversity to include video and 3D environments to further refine MLLM intelligence.
Summary updated