Abstract
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension.
Introduction: The Need for High-Ceiling Benchmarks
Machine Reading Comprehension (MRC) has seen a surge in interest, but many popular datasets used to train these models often have "low ceilings." Benchmarks like SQuAD or the CNN/Daily Mail cloze-style tasks frequently suffer from triviality, where answers are simple spans of text that models can "cheat" on using word-matching without true narrative understanding. Furthermore, automatically generated datasets often contain noise and a narrow topical scope, failing to test a model's ability to generalize across different writing styles.
The RACE dataset (ReAding Comprehension dataset from Examinations) was created to provide a "high-ceiling" benchmark. It forces models to engage in deeper reasoning, summarization, and attitude analysis—skills that human students are expected to master.
Dataset Construction and Methodology
The authors collected raw data from three large Chinese websites hosting English examinations for students aged 12-18. The resulting dataset is split into two subsets:
- RACE-M: Sourced from middle school exams (ages 12-15).
- RACE-H: Sourced from high school exams (ages 15-18).
In total, RACE contains 27,933 passages and 97,687 questions. The construction process involved rigorous filtering to ensure a four-option multiple-choice format and the removal of questions relying on non-textual elements like images. To evaluate the dataset, the authors utilized three distinct approaches: a rule-based Sliding Window baseline, the Stanford Attentive Reader, and the state-of-the-art Gated-Attention (GA) Reader.
Key Insights: Bridging the Reasoning Gap
A critical finding of this research is the "reasoning gap" between RACE and previous benchmarks. In RACE, nearly 60% of questions require either single-sentence or multi-sentence reasoning. This stands in stark contrast to datasets like SQuAD, where approximately 40% of questions can be solved by simple word matching.
Because the candidate answers were written by English instructors to challenge students, they often share similar word embeddings, preventing models from using simple elimination strategies. Furthermore, RACE introduces novel question types rarely seen in large-scale MRC, such as passage summarization (e.g., "What is the best title?") and attitude analysis.
Key Results: The Human-Machine Performance Gap
The results highlight a massive performance gap between humans and machines. While domain experts reached a ceiling performance of 94.5%, the best-performing model (the Gated-Attention Reader) achieved only 44.1% on the full dataset. On older datasets like CNN/Daily Mail, these same models achieve ~77-80%, which is very close to the human ceiling for those tasks. The failure of these models on RACE proves it is a much tougher "stress test" for Natural Language Understanding.
| Dataset | Random | Sliding Window | Stanford AR | GA Reader | Human (Ceiling) |
|---|---|---|---|---|---|
| RACE (Total) | 24.9% | 32.2% | 43.3% | 44.1% | 94.5% |
Conclusion and Future Directions
While RACE provides a challenging benchmark, it does have limitations, such as its ESL (English as a Second Language) context, which may use simpler language constructs than native literature. Additionally, many questions require external "world knowledge" or basic arithmetic that current neural architectures are not designed to handle. The authors conclude that current attention-based models are still largely performing sophisticated pattern matching; future work must focus on architectures capable of true information synthesis and logical deduction across multiple sentences.
Summary updated