Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

Amita Kamath, Jack Hessel, Khyathi Chandu, Jena D. Hwang, Kai-Wei Chang, Ranjay KrishnaarXiv 2026 · arXiv

TL;DR

Vision-Language Models (VLMs) consistently fail at basic spatial, temporal, counting, and negation reasoning because human-generated training data suffers from "reporting bias"—our natural tendency to omit obvious or tacit visual details. This paper demonstrates that scaling up models or datasets cannot fix this fundamental linguistic issue, but intentional, reasoning-aware annotator instructions can successfully bridge the gap.

What Problem Does This Paper Address?

Despite achieving impressive scores on standard benchmarks, state-of-the-art Vision-Language Models (VLMs) frequently stumble on seemingly trivial tasks. They struggle to count objects accurately, confuse spatial relationships ("left of" vs. "right of"), fail to track temporal order ("before" vs. "after"), and completely ignore negations ("is not flying").

The authors argue that this is not a fundamental flaw in model architecture, but rather a data problem caused by reporting bias. Grounded in the Gricean Maxims of pragmatics, humans communicate efficiently by omitting information that is contextually obvious or irrelevant. For example, a person is far more likely to caption a photo "at the game today!" than "a photo of 37 people standing behind a field." Because web-scale datasets are harvested from human-generated text, they systematically lack the explicit, tacit descriptions required to teach models these basic reasoning skills.

Methods

The authors investigate and attempt to mitigate reporting bias using a multi-pronged approach:

  1. Corpus Analysis: They performed keyword string searches across massive datasets to quantify the prevalence of reasoning language, manually verifying samples to calculate true positive rates.
  2. Scaling Law Evaluation: They evaluated 32 different OpenCLIP models across various data scales (up to LAION-2B) and compute budgets to determine if reasoning capabilities naturally emerge with scale.
  3. Controlled User Study: Using the Prolific platform, they tasked human annotators with captioning a fixed set of 100 images from MS COCO under four distinct instruction sets (COCO, LLaVA-1.5, PixMo, and their own custom instructions) to see if explicit prompts could override default communication biases.
  4. Finetuning Proof-of-Concept: They finetuned a LLaVA-1.5-13b model on a dataset enriched with counting data to verify if mitigating reporting bias directly translates to better model performance.

Datasets

  • Training & Analysis Corpora: LAION-2B, MS COCO, LLaVA-1.5 training set, and Molmo's training set (PixMo).
  • Evaluation Benchmarks: The authors curated and modified four existing datasets to isolate specific reasoning skills:
  • Spatial: Subset A of the What'sUp benchmark (412 balanced data points evaluating on, under, left of, right of).
  • Counting: A simplified version of CountBench (507 instances restricted to counts between 2–10).
  • Negations: A re-purposed version of the VAW benchmark (800 manually cleaned data points featuring templated negative attributes).
  • Temporal: The temporal relations subset of ControlledImCaps (200 balanced data points evaluating before and after).

Key Insights and Ideas

  • Scale Cannot Overcome Pragmatics: Increasing dataset size to billions of pairs or expanding model parameters does not trigger the "emergence" of reasoning. Because human communication habits do not change with scale, the percentage of reasoning-heavy instances remains flat and critically low (e.g., spatial reasoning is estimated to occur in only 0.1% of LAION-2B).
  • LLMs Inherit Human Bias: Synthetic data generated by LLMs (such as the LLaVA-1.5 instruction tuning set) suffers from the exact same reporting bias because these language models were originally trained on human text.
  • Keyword Presence $\neq$ Reasoning: Simply finding a keyword like "under" or "after" in a caption does not mean it operationalizes reasoning. For instance, the caption "Kitchen ideas under 5000" contains the word "under" but provides zero spatial utility (see Figure 1).
Figure 1: Examples from LAION-2B of data points that contain reasoning-related keywords that do and do not operationalize the reasoning capability itself.
  • Instructions Do Not Generalize: Prompting annotators to focus on one type of reasoning (like counting) successfully increases counting data, but does absolutely nothing to increase other types like negation or temporal language. Data curation instructions must be explicitly multi-faceted.

Key Results

  • The VLM Performance Gap: Open-source models fall an average of 54 points behind human performance on the curated reasoning benchmarks. Contrastive models (CLIP) perform near random chance on spatial and temporal tasks, and worse than random chance on negations (scoring ~11-13% vs. 25% random chance) because they tend to completely ignore the word "not".
Figure 2: Examples from our four benchmarks for contrastive and generative evaluations. The generative evalua- tion is in MCQ format but for counting, for which a free form output with a given range yielded higher scores.
  • User Study Impact: Custom instructions explicitly demanding all four reasoning types successfully raised the true occurrence of negation keywords from 2% (using COCO instructions) to 52%, and temporal keywords from 2% to 44% on the exact same set of images.
  • Finetuning Success: Finetuning LLaVA-1.5-13b on a dataset where counting data made up 39% of the corpus (matching the yield from their custom instructions) improved counting performance to 54.4%, outperforming both the base model (49.8%) and a baseline finetuned on standard LLaVA data (50.7%).

Limitations and Future Work

  • Resource Constraints: While the authors prove that reasoning-aware instructions work on a small scale (100 images) and via targeted finetuning, creating a web-scale pretraining dataset using these methods was beyond their organizational resources.
  • The Scaling Law Hypothesis: The conclusion that scale cannot fix reasoning relies on the extrapolation of current scaling trends, which could theoretically break or behave differently at drastically larger scales than those tested.
  • Future Directions: Promising next steps include automating the identification of data gaps in massive corpora, developing high-quality synthetic data pipelines that explicitly bypass reporting bias, and identifying natural communicative intents that inherently force the use of reasoning language.

Summary updated

Abstract

The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.
BibTeX
@misc{kamath2026scale,
  author = {Amita Kamath and Jack Hessel and Khyathi Chandu and Jena D. Hwang and Kai-Wei Chang and Ranjay Krishna},
  title = {{Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning}},
  year = {2026},
  eprint = {2602.23351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.23351},
}
Figure