Abstract
Reading Comprehension (RC) benchmarks often suffer from "guessability," where models exploit dataset biases to answer questions without true passage understanding. We present STARC (Structured Annotations for Reading Comprehension), a framework that enforces a principled structure for distractors and links them to specific text spans. Using this framework, we introduce OneStopQA, a high-quality dataset that exposes flaws in existing benchmarks like RACE and provides deeper diagnostic insights into model behavior.
Introduction
Progress in Natural Language Processing is frequently measured by performance on Reading Comprehension benchmarks. However, many popular datasets allow models to achieve high scores through simple pattern matching or by exploiting statistical biases—essentially "guessing" the answer. Furthermore, standard benchmarks typically only provide a binary correct/incorrect signal, offering little insight into why a model fails.
To address these issues, we propose the STARC framework. STARC moves beyond simple accuracy by requiring annotators to follow a rigorous structure for multiple-choice distractors and to identify the specific textual evidence (spans) required to answer or avoid "traps."
The STARC Framework
The core of STARC is a four-way multiple-choice structure designed to categorize model errors and ensure question quality:
- Type A (Correct): Requires comprehending a specific "critical span."
- Type B (Misunderstanding): A plausible but incorrect interpretation of the critical span.
- Type C (Irrelevant): Refers to a different part of the passage (a "distractor span") that does not answer the question.
- Type D (Unsupported): A plausible answer based on world knowledge but with no support in the text.
By requiring Auxiliary Span Annotations, STARC allows for systematic probing. Researchers can ablate the "critical span"; if a model still answers correctly, the question is deemed flawed or "guessable."
The OneStopQA Dataset
We applied the STARC framework to create OneStopQA, a dataset consisting of 1,458 paragraph-question pairs based on articles from The Guardian. Each article includes three parallel versions—Elementary, Intermediate, and Advanced—rewritten by professional editors. This multi-level structure allows for testing comprehension across varying linguistic complexities.
Experimental Results and Analysis
We compared OneStopQA against RACE, a widely used RC dataset. Our analysis revealed significant differences in robustness and quality.
The "Guessability" Gap
A major finding is the high "guessability" of the RACE dataset. When the passage is removed entirely, RoBERTa achieves 47.1% accuracy on RACE, far above the random baseline. In contrast, on OneStopQA, performance drops to 37.2%, indicating that the questions are much more dependent on actual passage comprehension.
Human-Machine Alignment
When analyzing errors, we found that both humans and state-of-the-art models like RoBERTa are most frequently misled by "Type B" distractors (misinterpretations of the correct span). This suggests that modern models are beginning to process text in a manner similar to humans, though they remain more susceptible to simpler biases.
Quality and Consistency
OneStopQA demonstrates superior clarity. Humans reached a performance ceiling of 97.9% on OneStopQA, compared to 88.8% on RACE. This suggests that the "difficulty" in older datasets often stems from ambiguity or poor question construction rather than complex reasoning requirements.
Conclusion
STARC provides a necessary diagnostic infrastructure for Reading Comprehension. While OneStopQA is smaller than some training corpora, its high-quality, structured nature makes it an ideal evaluation set. Future work includes using STARC to filter existing datasets and developing automated tools for generating high-quality educational assessments.
Summary updated