Multi-Matching Network for Multiple Choice Reading Comprehension

Min Tang, Jiaran Cai, Hankz Hankui ZhuoAAAI 2019 · DOI

Introduction

Traditional Machine Reading Comprehension (MRC) tasks, such as SQuAD, primarily focus on "span extraction," where the answer is a literal segment of the text. However, real-world comprehension—exemplified by standardized English examinations—requires sophisticated inference. The Multi-Matching Network (MMN) addresses this by mimicking human reasoning strategies: extracting evidence snippets based on questions and verifying "pseudo-statements" formed by combining questions with candidate answers.

Methodology

The MMN architecture is built on a "compose-match" framework designed to simulate two distinct human cognitive processes through three parallel modules:

  1. Evidence-Answer Matching (EAM-I and EAM-II): These modules mimic the process of locating evidence in a passage based on a question and then determining if that evidence supports a specific answer. They vary the order of attention (e.g., attending to the question first versus the answer) to refine evidence extraction.
  2. Question-Passage-Answer Matching (QPAM): This module simulates turning a question and answer into a "pseudo-statement" and checking if the passage entails that statement.
Overview of the Multi-Matching Network (MMN) architecture, featuring EAM and QPAM modules.

Technical Components

  • Gated Context Encoding: Utilizes a gated mechanism to control information flow from word embeddings to contextual representations.
  • SoftSel (Soft Selection): An attention-based operation that produces a weighted sum of one sequence relative to another to identify relevant passage segments.
  • Match Operation: A complex interaction layer that compares sequences using element-wise operations, followed by a BiGRU to aggregate matching information.
  • Merging Layer: Concatenates the outputs of EAM-I, EAM-II, and QPAM, allowing the final prediction layer to integrate different reasoning paradigms.

Experimental Setup

The model was evaluated on the RACE (Reading Comprehension Dataset from Examinations) dataset, which includes middle school (RACE-M) and high school (RACE-H) levels. RACE is significantly more difficult than span-extraction datasets because it is designed by human experts to test reasoning, often requiring synthesis of information not found verbatim in the text.

Results and Analysis

MMN established a new benchmark on the RACE dataset at the time of publication, achieving 54.7% overall accuracy for a single model and 58.2% for an ensemble. It outperformed strong baselines like BiAttention (MRU) and Hier-Co-Matching by approximately 4-5%.

Key Insights

  • QPAM Dominance: Ablation studies revealed that QPAM is the most critical component; its removal caused a 7.3% drop in accuracy, suggesting that triplet modeling (Passage-Question-Answer) is highly effective for multiple-choice formats.
  • Answer Length Correlation: The model performed better on questions with longer candidate answers. As shown in the performance trends, longer answers likely provide more semantic information for the matching layers to process.
Accuracy of MMN on RACE-M and RACE-H datasets relative to the length of candidate answers.
  • Question Type Performance: The model showed higher accuracy on "why" questions compared to "who" or "when" questions, likely because descriptive evidence is better suited for sequence-based matching than specific entity extraction.

Limitations and Conclusion

Despite its success, MMN struggles with negation (e.g., "true/false" or "not" questions), as sequence-matching architectures find it difficult to "flip" semantic polarity. Furthermore, a significant gap remains between machine performance and the human ceiling (94.5%). Future work involves incorporating commonsense knowledge and exploring planning-based approaches to enhance the interpretability of evidence extraction.

Summary updated

BibTeX
@inproceedings{tang2019multi,
  author = {Min Tang and Jiaran Cai and Hankz Hankui Zhuo},
  title = {{Multi-Matching Network for Multiple Choice Reading Comprehension}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33017088},
}
Figure