State-of-the-art language models can pass complex medical exams, write functional code, and synthesize dense academic literature. Yet, these same models frequently stumble on trivial questions: Is a cantaloupe rounder than a hammer? Do you need to open a door before walking through it? Does going on a vacation typically take longer than going for a walk?
This paradox stems from a fundamental mismatch between human communication and the data used to train machine learning models. This phenomenon is known as textual reporting bias.
The Inverse Frequency Paradox
To understand why language models fail at the obvious, we must look at why humans write. Human communication is driven by relevance and novelty. We document the unusual, the noteworthy, and the complex. We rarely, if ever, write down the self-evident rules of physical reality or social interaction because our audience already takes them for granted.
As Vered Shwartz observes in Unsupervised Commonsense Question Answering with Self-Talk, the frequency of information in a text corpus is often inversely proportional to its obviousness. A corpus might contain thousands of instances of the phrase "broken mirror" but zero instances stating that "mirrors are rigid objects made of glass."
Because language models learn purely from the statistical distributions of written text, this conversational omission creates a severe data sparsity problem. Michele Sebag argues in A tour of machine learning: An AI perspective that purely data-driven frequentist models struggle to acquire common sense precisely because human documentation systematically leaves this foundational knowledge unsaid. When a model's worldview is built entirely on text, it mistakes the unspoken for the untrue.
Dimensions of the Omission
Reporting bias distorts a model's understanding across several distinct domains of everyday knowledge:
Physical Properties and Affordances
Models excel at memorizing which actions frequently co-occur with specific objects in text, but they struggle to comprehend the underlying physical laws that govern those relationships. Ari Holtzman et al. demonstrate in Do Neural Language Representations Learn Physical Commonsense? that text-only models fail to link an object's implicit physical properties (e.g., rigidity) to its real-world affordances (e.g., can be shattered).
This creates a division between textual typicality and physical possibility. As explored by Su Wang et al. in Modeling Semantic Plausibility by Injecting World Knowledge, distributional data alone cannot teach a model to evaluate events that are physically plausible but never explicitly documented. Even when researchers attempt to elicit these properties from humans to build benchmarks, reporting bias persists; Dan Oneata notes in Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era that human-elicited feature norms still omit the most basic traits because participants take them for granted.
Temporal and Script Knowledge
Humans navigate the world using "scripts"—internalized sequences of the standard, mundane steps required to complete an activity (like ordering food at a restaurant). Because these intermediate steps are too boring to write down, language models struggle with zero-shot retrieval of standard event sequences. Abhilasha Sancheti et al. document this gap in What do Large Language Models Learn about Scripts?.
Similarly, relative durations of events are omitted from text. Ben Zhou et al. show in "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding that models lag significantly behind human performance when reasoning about the duration, frequency, and order of everyday occurrences.
Social and Causal Intuitions
In the social sphere, the most fundamental behavioral norms and internal mental states are rarely articulated. Hannah Rashkin et al. address this in Social IQa: Commonsense Reasoning about Social Interactions, showing that models lack exposure to the implicit rules governing human behavior. The authors of Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs argue that static text corpora act as a bottleneck, omitting basic social common sense because it is considered too obvious for a human audience.
This omission extends directly to causal reasoning. When models mine text for cause-and-effect relationships, they often miss mundane causal links. As Oktie Hassanzadeh et al. demonstrate in Answering Binary Causal Questions Through Large-Scale Text Mining: An Evaluation Using Cause-Effect Pairs from Human Experts, models fail on fundamental causal pairs that humans find trivial simply because authors find them unnecessary to document.
Why Scaling Alone Fails
A common assumption in deep learning is that scaling up the volume of training data will eventually fill any performance gaps. However, reporting bias is a structural property of human language, meaning that scraping more web text simply reinforces the same omissions.
If a fact is omitted from 1 billion pages, it is likely to be omitted from 100 billion pages. Instead of learning the missing foundations, larger models simply become more sophisticated at pattern matching the explicit, non-obvious text available to them. As Prajjwal Bhargava et al. point out in Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey, this creates a hard ceiling on a model's foundational understanding of the world, preventing true common sense from emerging solely from raw text scaling.
Bridging the Gap
To overcome reporting bias, researchers have shifted from relying purely on raw text corpora to developing explicit knowledge injection and elicitation frameworks.
[Raw Text Corpora] ──(Reporting Bias)──> [Gaps in Language Models]
▲
(Fills the Gap)
│
[Structured Knowledge Bases] ───(COMET/ECBRF)───────┘
Explicit Knowledge Graphs
One dominant strategy involves using structured, curated repositories of commonsense facts to supplement text models. Early approaches relied on static lookups in human-engineered databases like ConceptNet or Wikidata. However, modern techniques use language models themselves to generate and complete these graphs dynamically.
Antoine Bosselut et al. introduced this paradigm with COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. Instead of mining text for explicit mentions, COMET trains a model to generate implicit commonsense relations (like "if X happens, then Y wants to do Z") using seed data from structured graphs. This allows the model to extrapolate unwritten rules to entirely novel scenarios. Building on this, Zonglin Yang et al. developed the End-to-end Case-Based Reasoning for Commonsense Knowledge Base Completion framework, which uses analogical reasoning over similar past examples to complete missing commonsense links, bypassing the reporting bias inherent in standard language modeling data.
Latent Property Elicitation
Another approach focuses on extracting the implicit knowledge that models do manage to capture sub-symbolically, even if it is rarely generated out-of-the-box.
Yiben Yang et al. demonstrate in Extracting Commonsense Properties from Embeddings with Limited Human Guidance that by projecting standard word embeddings into a space defined by property "poles" (such as big vs. small), one can successfully extract relative physical properties without explicit textual statements.
Alternatively, models can be prompted to generate their own context before answering. Vered Shwartz's "Self-Talk" framework allows a model to uncover its own latent background knowledge by generating clarification questions and answers along its reasoning path. By forcing the model to explicitly articulate intermediate steps, these frameworks bridge the gap between what is written in the training data and what is required to reason about the real world.