A Machine Learning Approach to Answering Questions for Reading Comprehension Tests

Hwee Tou Ng, Leonghwee Teo, Jennifer Lai-Pheng KwanEMNLP 2000 · OpenReview

Abstract This paper explores the application of supervised machine learning to the task of reading comprehension (RC). By framing question answering as a classification problem and utilizing boosted decision trees, we demonstrate that automated systems can achieve performance levels competitive with manually engineered, rule-based systems. Our system, AQUAREAS, achieves a 39.3% accuracy on the Remedia dataset, significantly improving upon previous machine learning attempts.

Introduction

The Reading Comprehension (RC) task requires a system to "understand" a short narrative well enough to answer specific questions (who, what, when, where, why). Unlike Information Retrieval (IR), which searches for answers across massive document collections, RC focuses on deep processing of a single text. At the time of this research, the prevailing view was that RC required complex, handcrafted deterministic rules due to the "uncooperative" nature of the data. Previous machine learning (ML) attempts had yielded poor results (approx. 14% accuracy). This study aims to demonstrate that with appropriate feature engineering and ensemble learning, ML is a robust and competitive alternative.

The AQUAREAS System

The proposed system, AQUAREAS (Automated QUestion Answering upon REAding Stories), models the RC task as a sentence classification problem. Every sentence in a story is treated as a candidate answer. For each question, the system generates a feature vector for every sentence and uses a classifier to predict the likelihood of that sentence being the correct answer.

Feature Engineering

AQUAREAS utilizes 20 features to capture linguistic and structural cues:

  • Relative Matching Scores (DMWM/DMVM): Instead of absolute scores, the system uses the Difference-from-Max-Word-Match (DMWM) and Difference-from-Max-Verb-Match (DMVM). These measure how well a sentence matches the question relative to the best-matching sentence in the same story.
  • Named Entity Recognition (NER) and Coreference: The system identifies entities (Person, Location, Date, etc.) and uses coreference chains to propagate these tags, ensuring that pronouns (e.g., "he") are correctly associated with their antecedents.
  • Automated Keyword Discovery: Using a statistical correlation metric ($C$-value), the system automatically identifies "clue words" from the training data (e.g., "year" for when questions), reducing the need for manual rule creation.

Learning Algorithm

The core of AQUAREAS is the C5 decision tree algorithm enhanced with adaptive boosting. Boosting allows the system to generate a sequence of trees, where each subsequent tree focuses on the instances misclassified by its predecessors.

Figure 2: A sequence of five boosted decision trees generated for 'When' questions.

As shown in Figure 2, the boosted ensemble for "When" questions incorporates various features such as relative match scores and the presence of date-related keywords or entities (e.g., Sent-contains-Date, Year-keyword-in-Sent).

Experimental Evaluation

The system was evaluated using the Remedia corpus, consisting of 115 children's stories written at grade levels two through five. The dataset was split into 55 training stories and 60 test stories. Performance was measured using the HumSent metric, which calculates the percentage of questions for which the system's top-ranked sentence matches a human-judged correct answer.

Results and Analysis

AQUAREAS achieved an overall HumSent accuracy of 39.3%. This performance is highly competitive with the leading rule-based systems of the era, such as Deep Read (36.3%) and Quarc (39.7%), and represents a nearly three-fold improvement over previous ML-based attempts.

Ablation studies revealed that word matching remains the most critical feature across all question types. Coreference resolution was particularly beneficial for who, when, and where questions, where identifying the specific entities involved is paramount. However, "why" questions remained a significant challenge, with the system achieving only 13.3% accuracy, likely due to the need for complex causal reasoning beyond the scope of current feature sets.

Conclusion

This research demonstrates that machine learning is a viable and effective approach for reading comprehension tasks. By leveraging relative matching scores and automated feature discovery, AQUAREAS matches the performance of labor-intensive rule-based systems. Future work will focus on incorporating more sophisticated knowledge sources to address complex question types like "why" and "how."

Summary updated

BibTeX
@inproceedings{ng2000machine,
  author = {Hwee Tou Ng and Leonghwee Teo and Jennifer Lai-Pheng Kwan},
  title = {{A Machine Learning Approach to Answering Questions for Reading Comprehension Tests}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2000},
  url = {https://openreview.net/forum?id=W222QL1Vssu},
}
Figure