Introduction
Reading Comprehension (RC) is a complex cognitive task that integrates various linguistic and reasoning abilities. While the field has seen significant progress in improving accuracy on benchmarks like MCTest and SQuAD, evaluation often remains a "black-box" process. A single accuracy score fails to reveal why a system succeeds or fails—whether it struggles with pronoun resolution, causal links, or basic world knowledge.
Sugawara et al. (2017) address this by proposing a "unit testing" methodology for RC. By decomposing the task into specific prerequisite skills, they provide a granular analysis of both dataset difficulty and system performance, moving beyond aggregate metrics to diagnose specific weaknesses in current models.
Defining Prerequisite Skills
The authors identify ten core skills required for successful reading comprehension, drawing from established NLP tasks. These skills range from low-level linguistic processing to high-level reasoning.
These skills include tracking entities (List/Enumeration), resolving pronouns (Coreference Resolution), and understanding logical connectors or spatiotemporal relations. Notably, skills like Commonsense Reasoning and Causal Relations represent significant hurdles for automated systems, as they often require information not explicitly stated in the text.
Methodology and Dataset
The study utilizes the MCTest dataset (specifically the MC160 and MC500 subsets), which consists of fictional stories written for children. MCTest is an ideal testbed because its limited vocabulary ensures that difficulty arises from reasoning requirements rather than lexical complexity.
The researchers manually annotated MCTest questions with the prerequisite skills needed to answer them. They then evaluated four distinct RC systems to see how they performed on each specific skill:
- Baseline SW+D: A sliding-window and distance-based baseline.
- Smith No RTE / RTE: Feature-based models.
- Yin ABCNN: A neural network-based model (Attention-Based Convolutional Neural Network).
Results and Analysis
The analysis reveals that RC is rarely a single-skill endeavor. Approximately 61.3% of questions in the dataset require multiple skills, with an average of 1.94 skills per question. This multi-faceted nature explains the inherent difficulty of the task compared to isolated NLP challenges.
As shown in Table 2, Coreference Resolution (63.8%) and Commonsense Reasoning (41.9%) are the most frequently required skills in MCTest. System performance varies wildly across categories. While some models perform well on logical reasoning, they often struggle with mathematical operations and causal relations, where accuracy can drop as low as 20%.
A critical finding is the inverse correlation between complexity and accuracy. As the number of required skills increases, performance drops across all systems. For instance, accuracy on questions requiring zero skills (simple matching) ranges from 55-75%, but plummets to 33-42% for questions requiring five skills. Interestingly, the neural ABCNN model did not show a significant "reasoning leap" over feature-based baselines, though its performance remained more stable across varying skill counts, suggesting a more holistic (if not necessarily more accurate) processing style.
Conclusion
This research highlights the necessity of granular evaluation in Reading Comprehension. By identifying and annotating prerequisite skills, the authors demonstrate that current systems—including neural models—still face significant challenges with complex reasoning and commonsense knowledge. Future work aims to automate the generation of these "unit tests" to help researchers systematically stress-test and improve RC models.
Reference: Sugawara, S., Yokono, H., & Aizawa, A. (2017). Prerequisite Skills for Reading Comprehension: Multi-Perspective Analysis of MCTest Datasets and Systems. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17).
Summary updated