STARC: Structured Annotations for Reading Comprehension

Yevgeni Berzak, Jonathan Malmaud, Roger Levy

Multiple-choice reading comprehension (RC) is a cornerstone of NLP evaluation, yet it faces a significant challenge: "gaming" the system. Many popular datasets, such as RACE, contain biases that allow models to achieve high accuracy using simple heuristics—like word matching or statistical shortcuts—without truly processing the text. Furthermore, standard benchmarks provide a single accuracy score that fails to explain why a model fails.

This paper, published at ACL 2020, introduces STARC (Structured Annotations for Reading Comprehension), a framework designed to enforce genuine reading and provide granular diagnostic insights into model behavior.

The STARC Framework

The STARC framework moves beyond simple question-answer pairs by introducing two major structural constraints:

  1. Structured Answer Choices: Instead of random distractors, every question features four specific types:
  2. (A) Correct: A paraphrase of the information.
  3. (B) Misunderstanding: A plausible but incorrect interpretation of the same text used for the correct answer.
  4. (C) Irrelevant: Information present in the passage that does not answer the specific question.
  5. (D) No Support: A plausible-sounding answer with no basis in the text.
  6. Auxiliary Span Annotations: Answers are explicitly linked to a "critical span" (the text required to answer) or a "distractor span" (text that might lead to picking choice C).
Example of STARC annotations showing difficulty levels and structured answer choices.

OneStopQA: A New Benchmark

To validate STARC, the authors released OneStopQA, a dataset of 1,458 paragraph-question pairs based on Guardian articles. Each article is provided in three difficulty levels (Elementary, Intermediate, and Advanced), rewritten by professional editors. This allows researchers to test how models handle varying linguistic complexity while keeping the underlying logic constant.

Analyzing the "Guessability Gap"

A key contribution of this work is the use of Ablation-based Data Quality Probing. By systematically removing the passage or the question, the authors measured how much a model relies on genuine comprehension versus shallow patterns.

The results revealed a striking "guessability gap" in existing benchmarks. As shown in the performance table below, a RoBERTa model can guess the correct answer to 47.1% of RACE questions without even looking at the passage. In contrast, on OneStopQA, the "no-passage" accuracy drops to 37.2%, significantly closer to the random baseline of 25%.

Performance and ablation results for RoBERTa and humans on RACE and OneStopQA.

Key Insights and Results

  • Human Ceiling: OneStopQA offers a much higher evaluation ceiling. Human accuracy on OneStopQA is 95.3%, compared to 84.7% on RACE. This is largely because OneStopQA is cleaner; humans found 18.3% of RACE questions to be invalid (no unique correct answer), while the invalid rate for OneStopQA was only 3.4%.
  • Models Fail Like Humans: The A/B/C/D structure revealed that both humans and state-of-the-art models like RoBERTa share a similar error distribution, most frequently tripped up by "Type B" errors (misunderstanding the correct span).
  • Span Validation: When the "critical span" of text was removed, model performance plummeted to near-guessing levels (41.1%). This confirms that the STARC framework successfully identifies the exact text required for comprehension.
  • Heuristic Resistance: Simpler models, such as the Stanford Attentive Reader, saw a massive performance drop on OneStopQA compared to RACE (from 42.8% to 34.3%), suggesting the new dataset is more resistant to simple word-matching tricks.

Conclusion

While OneStopQA is smaller than massive training sets like SQuAD, it serves as a high-quality diagnostic tool. The STARC framework demonstrates that by structuring distractors and mapping them to specific text spans, we can build more robust evaluations that distinguish between shallow pattern matching and true reading comprehension. Future work may involve using STARC to automatically flag and improve "guessable" questions in larger datasets.

Summary updated

BibTeX
@inproceedings{berzak2020starc,
  author = {Yevgeni Berzak and Jonathan Malmaud and Roger Levy},
  title = {{STARC: Structured Annotations for Reading Comprehension}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.507},
}
Figure