TL;DR
Vision-Language Models (VLMs) consistently fail at basic spatial, temporal, counting, and negation reasoning because human-generated training data suffers from "reporting bias"—our natural tendency to omit obvious or tacit visual details. This paper demonstrates that scaling up models or datasets cannot fix this fundamental linguistic issue, but intentional, reasoning-aware annotator instructions can successfully bridge the gap.
What Problem Does This Paper Address?
Despite achieving impressive scores on standard benchmarks, state-of-the-art Vision-Language Models (VLMs) frequently stumble on seemingly trivial tasks. They struggle to count objects accurately, confuse spatial relationships ("left of" vs. "right of"), fail to track temporal order ("before" vs. "after"), and completely ignore negations ("is not flying").
The authors argue that this is not a fundamental flaw in model architecture, but rather a data problem caused by reporting bias. Grounded in the Gricean Maxims of pragmatics, humans communicate efficiently by omitting information that is contextually obvious or irrelevant. For example, a person is far more likely to caption a photo "at the game today!" than "a photo of 37 people standing behind a field." Because web-scale datasets are harvested from human-generated text, they systematically lack the explicit, tacit descriptions required to teach models these basic reasoning skills.
Methods
The authors investigate and attempt to mitigate reporting bias using a multi-pronged approach:
- Corpus Analysis: They performed keyword string searches across massive datasets to quantify the prevalence of reasoning language, manually verifying samples to calculate true positive rates.
- Scaling Law Evaluation: They evaluated 32 different OpenCLIP models across various data scales (up to LAION-2B) and compute budgets to determine if reasoning capabilities naturally emerge with scale.
- Controlled User Study: Using the Prolific platform, they tasked human annotators with captioning a fixed set of 100 images from MS COCO under four distinct instruction sets (COCO, LLaVA-1.5, PixMo, and their own custom instructions) to see if explicit prompts could override default communication biases.
- Finetuning Proof-of-Concept: They finetuned a LLaVA-1.5-13b model on a dataset enriched with counting data to verify if mitigating reporting bias directly translates to better model performance.
Datasets
- Training & Analysis Corpora: LAION-2B, MS COCO, LLaVA-1.5 training set, and Molmo's training set (PixMo).
- Evaluation Benchmarks: The authors curated and modified four existing datasets to isolate specific reasoning skills:
- Spatial: Subset A of the What'sUp benchmark (412 balanced data points evaluating on, under, left of, right of).
- Counting: A simplified version of CountBench (507 instances restricted to counts between 2–10).
- Negations: A re-purposed version of the VAW benchmark (800 manually cleaned data points featuring templated negative attributes).
- Temporal: The temporal relations subset of ControlledImCaps (200 balanced data points evaluating before and after).
Key Insights and Ideas
- Scale Cannot Overcome Pragmatics: Increasing dataset size to billions of pairs or expanding model parameters does not trigger the "emergence" of reasoning. Because human communication habits do not change with scale, the percentage of reasoning-heavy instances remains flat and critically low (e.g., spatial reasoning is estimated to occur in only 0.1% of LAION-2B).
- LLMs Inherit Human Bias: Synthetic data generated by LLMs (such as the LLaVA-1.5 instruction tuning set) suffers from the exact same reporting bias because these language models were originally trained on human text.
- Keyword Presence $\neq$ Reasoning: Simply finding a keyword like "under" or "after" in a caption does not mean it operationalizes reasoning. For instance, the caption "Kitchen ideas under 5000" contains the word "under" but provides zero spatial utility (see Figure 1).
- Instructions Do Not Generalize: Prompting annotators to focus on one type of reasoning (like counting) successfully increases counting data, but does absolutely nothing to increase other types like negation or temporal language. Data curation instructions must be explicitly multi-faceted.
Key Results
- The VLM Performance Gap: Open-source models fall an average of 54 points behind human performance on the curated reasoning benchmarks. Contrastive models (CLIP) perform near random chance on spatial and temporal tasks, and worse than random chance on negations (scoring ~11-13% vs. 25% random chance) because they tend to completely ignore the word "not".
- User Study Impact: Custom instructions explicitly demanding all four reasoning types successfully raised the true occurrence of negation keywords from 2% (using COCO instructions) to 52%, and temporal keywords from 2% to 44% on the exact same set of images.
- Finetuning Success: Finetuning LLaVA-1.5-13b on a dataset where counting data made up 39% of the corpus (matching the yield from their custom instructions) improved counting performance to 54.4%, outperforming both the base model (49.8%) and a baseline finetuned on standard LLaVA data (50.7%).
Limitations and Future Work
- Resource Constraints: While the authors prove that reasoning-aware instructions work on a small scale (100 images) and via targeted finetuning, creating a web-scale pretraining dataset using these methods was beyond their organizational resources.
- The Scaling Law Hypothesis: The conclusion that scale cannot fix reasoning relies on the extrapolation of current scaling trends, which could theoretically break or behave differently at drastically larger scales than those tested.
- Future Directions: Promising next steps include automating the identification of data gaps in massive corpora, developing high-quality synthetic data pipelines that explicitly bypass reporting bias, and identifying natural communicative intents that inherently force the use of reasoning language.
Summary updated