Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Introduction
In the quest to build agents that understand text as well as humans do, the community relies on datasets to train and evaluate models. However, researchers argue that existing benchmarks often suffer from three critical flaws: triviality (relying on simple span-based answers), noise (low human ceiling due to poor quality), and narrow scope (limited domains).
To address these issues, researchers from Carnegie Mellon University have introduced RACE, a massive reading comprehension dataset consisting of nearly 100,000 questions sourced from English exams for Chinese middle and high school students. Unlike previous benchmarks where machines have already neared human performance, RACE reveals a staggering 50% gap between state-of-the-art models and human experts, making it a new "gold standard" for testing true machine reasoning.
The RACE Dataset
RACE (ReAding Comprehension from Examinations) is split into two subsets: RACE-M (middle school, ages 12-15) and RACE-H (high school, ages 15-18). In total, the dataset contains 27,933 passages and 97,687 questions.
A key differentiator of RACE is its expert-designed complexity. Because these questions were originally written by English instructors to test human students, they include "distractors" that appear to be word-matches but are contextually incorrect. As shown in the comparison below, RACE significantly emphasizes reasoning over simple pattern matching.
Nearly 60% of RACE questions require single- or multi-sentence reasoning, whereas 39.8% of SQuAD questions are simple word matches. RACE also introduces "whole-picture reasoning" and "attitude analysis," which are largely absent from other large-scale datasets.
Methods
To establish a baseline, the authors evaluated several high-performing architectures:
- Sliding Window Algorithm: A rule-based baseline that calculates TF-IDF style matching scores between the question/answer pair and the document.
- Stanford Attentive Reader (SAR): A neural model using bidirectional GRUs and a bilinear attention mechanism.
- Gated-Attention Reader (GA): A more complex multi-hop architecture that scans the document and question iteratively to refine its internal representation.
Human evaluations were conducted using both Amazon Mechanical Turk workers ("Turkers") and expert human instructors to establish a performance ceiling.
Results and Analysis
The quantitative results highlight the difficulty of the new benchmark. The Gated-Attention (GA) reader achieved the best result with 44.1% accuracy on the full RACE set. In contrast, Amazon Turkers achieved 73.3%, and the human expert "ceiling" reached 94.5%.
Simple word-matching (Sliding Window) fails significantly on RACE, scoring only 32.2%. This proves that the dataset successfully resists "dumb" search strategies. There is also a clear difficulty curve; models performed better on the middle school subset (RACE-M) than on the high school subset (RACE-H), where the Sliding Window baseline dropped to 30.4%.
Conclusion
RACE exposes a significant "reasoning gap" in current NLP models. While machines are within 5-10% of human performance on many datasets, the ~50% gap on RACE suggests that current models are still "literate" only in a superficial sense. Future work must focus on developing architectures capable of multi-sentence synthesis and abstractive reasoning to bridge this gap.
Summary updated