If you ask a modern natural language processing model who signed a historic treaty, where an event took place, or when a specific disaster occurred, it will likely answer with remarkable precision. But if you ask that same model why the treaty was signed or how the disaster led to subsequent socio-economic shifts, its performance drops precipitously.
This persistent asymmetry is known as the causal reasoning gap in question answering (QA). Across decades of research—spanning early rule-based systems, feature-engineered machine learning, and deep neural architectures—automated systems have consistently excelled at extracting explicit facts while struggling to synthesize explanations. Understanding why this gap exists requires looking closely at the mechanics of how models read and where those mechanics break down.
The Shortcut of Factoid Alignment
To understand why causal queries fail, we must first understand why factual queries succeed. Traditional factoid question answering relies heavily on surface alignment and entity type matching.
When a system processes a query starting with "Who," "Where," or "When," it immediately narrows its search space to specific token classes: names, locations, or dates. As demonstrated by Helen M. Meng et al. in The Use of Metadata, Web-derived Answer Patterns and Passage Context to Improve Reading Comprehension Performance, leveraging structural patterns and metadata weighting is highly effective for isolating these bounded, explicit pieces of information. The task is essentially reduced to a sophisticated form of pattern matching: finding the semantic overlap between the question and a candidate sentence containing the correct entity type.
Why Causal Queries Break the Paradigm
Causal and explanatory questions (typically triggered by "why" or "how") completely bypass these shortcuts. They introduce three fundamental challenges that surface-level matching cannot resolve:
1. The Absence of a Target Entity Type
There is no single grammatical category or entity tag for a "reason." An explanation can be a verb phrase, a subordinate clause, a full sentence, or an entire paragraph. As Jong-Hoon Oh et al. observe in Why Question Answering using Sentiment Analysis and Word Classes, "why" questions demand underlying logic and motivations rather than discrete dates or names, leaving models without a clear structural anchor in the text.
2. Rhetorical and Implicit Structures
In natural language, the relationship between a cause and an effect is rarely spelled out with explicit markers like "because." More often, causality is embedded within the narrative flow or rhetorical structure of a text. Early foundational work by Lynette Hirschman et al. on Deep Read: A Reading Comprehension System highlighted that automated systems struggle with these queries because they require interpreting implicit relationships that simple sentence-level matching completely misses.
3. The Failure of Span Extraction
Modern QA architectures are heavily optimized for span extraction—identifying a contiguous string of text within a passage to serve as the answer. However, as Hwee Tou Ng et al. argue in Question Answering Using a Large Text Database: A Machine Learning Approach, extractive methods focusing on short noun phrases are fundamentally unsuited for complex queries that require multi-passage synthesis or full-sentence explanations.
Evidence Across Generations of Architectures
The explanatory gap is not an artifact of any single era of machine learning; it is a systemic limitation that has persisted as architectures have evolved.
[Factoid Queries] ---> Surface Alignment & Entity Matching ---> High Accuracy
[Causal Queries] ---> Requires Logical Synthesis & Rhetorical Parsing ---> Performance Drop
- Feature-Based and Early ML Models: Early statistical approaches quickly hit a ceiling when dealing with explanatory logic. In their evaluation of reading comprehension tests, Hwee Tou Ng et al. showed in A Machine Learning Approach to Answering Questions for Reading Comprehension Tests that feature-based classification models fail on causal reasoning because surface-level cues like word matching and entity tagging cannot capture underlying explanatory logic.
- Deep Learning and Coattention Networks: The advent of deep neural networks did not automatically solve the problem. High-performing architectures like the Dynamic Coattention Network, introduced by Caiming Xiong et al. in Dynamic Coattention Networks For Question Answering, demonstrated a stark performance divide: they achieved high proficiency at extracting factual entities like dates and locations, but frequently failed to synthesize the logical explanations required for "why" questions.
- Span-Prediction Bottlenecks: Popular models of the late 2010s, such as R-NET, exposed the limits of span-extraction techniques. As Wenhui Wang et al. noted in Gated Self-Matching Networks for Reading Comprehension and Question Answering, accuracy decreases sharply as answer complexity increases. Similarly, the match-LSTM framework developed by Shuohang Wang et al. in Machine Comprehension Using Match-LSTM and Answer Pointer confirmed that the linguistic diversity of causal explanations makes them significantly less grounded in direct lexical or syntactic patterns.
- Attention and Retrieval Limits: Even fully parallelized, attention-only architectures struggle to bridge this gap. The authors of A Fully Attention-Based Information Retriever, led by Alvaro H. C. Correia, found that while attention mechanisms excel at identifying factual entities, they often lack the explicit mechanisms required to synthesize multi-step explanations. When scaled to massive corpora, dual encoder retrieval models face a similar hurdle; Amin Ahmad et al. point out in ReQA: An Evaluation for End-to-End Answer Retrieval Models that these models rely on semantic matching rather than the deep logical reasoning necessary to resolve complex causal queries.
A Universal Challenge Across Modalities and Languages
The causal reasoning gap is not confined to English text comprehension; it reappears wherever models are tasked with understanding relationships rather than attributes.
Structured Knowledge Bases
Knowledge Base Question Answering (KBQA) systems are highly effective at retrieving attribute-based facts (such as a patient's medication dosage). However, as Ping Wang 0024 et al. demonstrate in Attention-based aspect reasoning for knowledge base question answering on clinical notes, these systems struggle to explain the underlying mechanisms or causal relationships behind clinical observations. This stems from a foundational design choice: as Atsushi Fujii et al. observe in Oganizing Encyclopedic Knowledge based on the Web and its Application to Question Answering, most knowledge organization frameworks are optimized for definitional queries rather than procedural or causal sequences.
Video and Narrative Understanding
When moving from text to multimodal environments, the gap widens. Benchmarks like MovieQA, introduced by Makarand Tapaswi et al. in MovieQA: Understanding Stories in Movies through Question-Answering, explicitly target narrative logic by asking "why" and "how" events occur across video and scripts. In a comprehensive review of the field, Wei Ji et al. note in Video Question Answering: Datasets, Algorithms and Challenges that a massive performance disparity remains between simple recognition tasks (identifying objects or actions) and complex causal or temporal reasoning (explaining the underlying motivations behind those actions).
Situational Awareness and Low-Resource Settings
Whether analyzing social media data for disaster response (Kiyonori Ohtake et al. in NICT Disaster Information Analysis System) or evaluating low-resource language benchmarks (Vésteinn Snæbjarnarson et al. in Natural Questions in Icelandic), the pattern remains invariant. Models consistently provide strong situational awareness when describing what is happening, but falter when asked to explain the underlying causes.
The Mirror Problem: Causal Question Generation
This bottleneck also manifests in the inverse task: Neural Question Generation (NQG). If a model cannot reason about causality, it cannot generate meaningful questions about it.
When training models to generate questions from text, systems easily master factoid generation by identifying an entity and applying local grammatical heuristics. However, as Shijie Zhang et al. show in Automatic Generation of Grounded Visual Questions, generating natural and logical "why" and "how" questions requires a deeper grasp of causal and procedural logic that local heuristics cannot capture. Furthermore, Nan Yang et al. point out in Neural Question Generation from Text: A Preliminary Study that these complex query types are frequently underrepresented in training data, compounding the difficulty for models attempting to learn the subtle, non-local relationships that define human inquiry.
The consensus across decades of QA research points to a clear conclusion: closing the causal reasoning gap requires moving beyond architectures that excel merely at semantic alignment and span extraction. True comprehension demands mechanisms explicitly designed to parse, map, and synthesize the multi-step logical structures of cause and effect.
Key Foundational Work
A Machine Learning Approach to Answering Questions for Reading Comprehension Tests (2000) — contributed 1 concept elements to this direction.
Deep Read: A Reading Comprehension System (1999) — contributed 1 concept elements to this direction.