STARC: Structured Annotations for Reading Comprehension

Yevgeni Berzak, Jonathan Malmaud, Roger Levy

Abstract

Reading Comprehension (RC) benchmarks often suffer from "guessability," where models exploit dataset biases to answer questions without true passage understanding. We present STARC (Structured Annotations for Reading Comprehension), a framework that enforces a principled structure for distractors and links them to specific text spans. Using this framework, we introduce OneStopQA, a high-quality dataset that exposes flaws in existing benchmarks like RACE and provides deeper diagnostic insights into model behavior.

Introduction

Progress in Natural Language Processing is frequently measured by performance on Reading Comprehension benchmarks. However, many popular datasets allow models to achieve high scores through simple pattern matching or by exploiting statistical biases—essentially "guessing" the answer. Furthermore, standard benchmarks typically only provide a binary correct/incorrect signal, offering little insight into why a model fails.

To address these issues, we propose the STARC framework. STARC moves beyond simple accuracy by requiring annotators to follow a rigorous structure for multiple-choice distractors and to identify the specific textual evidence (spans) required to answer or avoid "traps."

The STARC Framework

The core of STARC is a four-way multiple-choice structure designed to categorize model errors and ensure question quality:

  • Type A (Correct): Requires comprehending a specific "critical span."
  • Type B (Misunderstanding): A plausible but incorrect interpretation of the critical span.
  • Type C (Irrelevant): Refers to a different part of the passage (a "distractor span") that does not answer the question.
  • Type D (Unsupported): A plausible answer based on world knowledge but with no support in the text.
Example of STARC annotations showing parallel text versions and structured distractors.

By requiring Auxiliary Span Annotations, STARC allows for systematic probing. Researchers can ablate the "critical span"; if a model still answers correctly, the question is deemed flawed or "guessable."

The OneStopQA Dataset

We applied the STARC framework to create OneStopQA, a dataset consisting of 1,458 paragraph-question pairs based on articles from The Guardian. Each article includes three parallel versions—Elementary, Intermediate, and Advanced—rewritten by professional editors. This multi-level structure allows for testing comprehension across varying linguistic complexities.

Experimental Results and Analysis

We compared OneStopQA against RACE, a widely used RC dataset. Our analysis revealed significant differences in robustness and quality.

The "Guessability" Gap

A major finding is the high "guessability" of the RACE dataset. When the passage is removed entirely, RoBERTa achieves 47.1% accuracy on RACE, far above the random baseline. In contrast, on OneStopQA, performance drops to 37.2%, indicating that the questions are much more dependent on actual passage comprehension.

Performance of RoBERTa and humans on RACE vs. OneStopQA under various ablation conditions.

Human-Machine Alignment

When analyzing errors, we found that both humans and state-of-the-art models like RoBERTa are most frequently misled by "Type B" distractors (misinterpretations of the correct span). This suggests that modern models are beginning to process text in a manner similar to humans, though they remain more susceptible to simpler biases.

Quality and Consistency

OneStopQA demonstrates superior clarity. Humans reached a performance ceiling of 97.9% on OneStopQA, compared to 88.8% on RACE. This suggests that the "difficulty" in older datasets often stems from ambiguity or poor question construction rather than complex reasoning requirements.

Conclusion

STARC provides a necessary diagnostic infrastructure for Reading Comprehension. While OneStopQA is smaller than some training corpora, its high-quality, structured nature makes it an ideal evaluation set. Future work includes using STARC to filter existing datasets and developing automated tools for generating high-quality educational assessments.

Summary updated

BibTeX
@inproceedings{berzak2020starc,
  author = {Yevgeni Berzak and Jonathan Malmaud and Roger Levy},
  title = {{STARC: Structured Annotations for Reading Comprehension}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.507},
}
Figure