Machine reading comprehension (MRC) models are becoming increasingly sophisticated, yet they exhibit a fundamental cognitive flaw: belief bias. This phenomenon, well-documented in human psychology, occurs when a model prioritizes its internal "world knowledge" over the specific evidence provided in a text. This paper introduces AUTO-LOCKE, a challenge dataset designed to expose this bias by presenting models with simple logic puzzles that contradict common facts.
The Problem: Priors Overriding Context
In psychology, belief bias is the tendency to judge the validity of an argument based on how believable the conclusion is, rather than its logical consistency. For instance, if a person is told "All fish can fly; a salmon is a fish," they may hesitate to conclude "a salmon can fly" because it contradicts reality.
Modern MRC systems, pre-trained on massive corpora like Wikipedia, develop strong internal beliefs. When faced with fictional or counter-factual contexts—such as a story where "Washington is a number"—these models often ignore the provided text and revert to their pre-trained facts (e.g., answering that Washington is a city). This brittleness poses significant risks for applications in creative writing, fiction, or rapidly evolving domains where facts change.
Methodology: The AUTO-LOCKE Dataset
To systematically test for belief bias, the author developed LOCKE’S HOLIDAY, a set of hand-crafted templates and real-world examples from social media where the context is intentionally nonsensical or counter-factual.
To scale this evaluation, the author created AUTO-LOCKE, a synthetic dataset generator. By taking 20 core templates and using WordNet and GloVe embeddings to swap entities with their "nearest neighbors," the generator produced 11,699 variations. These examples are logically trivial but factually "weird," forcing the model to choose between the provided context and its internal priors. A human baseline study confirmed these tasks are effortless for people, who achieved 100% accuracy.
Experimental Setup
The study evaluates four distinct machine reading architectures:
- BiDAF: A standard recurrent neural network model.
- ELMo-BiDAF: An iteration using contextualized word embeddings.
- TransformerQA: A high-performing model based on RoBERTa.
- NAQANet: A model specialized for numerical reasoning.
These models were trained on standard benchmarks like SQuAD 2.0 and DROP before being tested on the AUTO-LOCKE challenge set.
Key Results: The "Better" Model Paradox
The most striking finding is that models which excel on standard benchmarks often perform the worst on AUTO-LOCKE. This suggests that as models become "smarter" and more knowledgeable through pre-training, they become more stubborn and prone to belief bias.
As shown in the results, while TransformerQA significantly outperforms ELMo-BiDAF on SQuAD 2.0 (by nearly 30 F1 points), the older ELMo-BiDAF model is actually more robust on the AUTO-LOCKE Exact Match (EM) metric. This "rank reversal" indicates that the stronger language priors in Transformer-based models make them less likely to listen to the provided context when it conflicts with their training data.
Discussion and Future Work
The failure of these models to handle simple counter-factuals like "Washington is a number" represents a catastrophic failure of trust. If a model cannot be relied upon to follow explicit instructions in a short context, its utility in complex, real-world reasoning is limited.
The paper suggests that simply scaling up language models may exacerbate this bias. Instead, future research should focus on "hybrid" architectures that can effectively disentangle general world knowledge from the ability to perform abstract, logical reasoning. Potential solutions include training on fictional texts or employing adversarial data augmentation to teach models that the provided context is always the primary source of truth.
Summary updated