Learning to Classify the Wrong Answers for Multiple Choice Question Answering (Student Abstract)

Hyeondey Kim, Pascale FungAAAI 2020 · DOI

Abstract

Multiple Choice Question Answering (MCQA) remains a challenging task for machine reading comprehension models, often requiring complex reasoning to distinguish between plausible distractors and the correct answer. This paper proposes the Wrong Answer Ensemble (WAE), a model-agnostic framework that mimics human elimination strategies by explicitly training models to identify incorrect options. By integrating a "Wrong Answer" (WA) model with a standard "Correct Answer" (CA) model, we achieve significant performance gains on the DREAM dataset. Specifically, our BERT-large WAE variant reaches 69.0% accuracy, a 2.7% improvement over the baseline.

Introduction

In standard Machine Reading Comprehension (MRC) tasks like SQuAD, models primarily focus on extracting surface-level information. However, MCQA in an examination context is significantly more difficult because it requires reasoning through "distractors"—options designed to look plausible but which are technically incorrect.

Current models typically focus on finding the "needle in the haystack" (the correct answer) without learning the characteristics of the "hay" (the wrong answers). In contrast, humans often solve such problems through elimination: identifying and discarding unlikely options to narrow the field. This research bridges this gap by teaching models to recognize distractors, effectively adopting an elimination-style strategy for automated test-taking.

Methods

The proposed Wrong Answer Ensemble (WAE) is a framework that can be layered onto existing transformer-based architectures.

Architecture and Training

The system utilizes BERT as a backbone, fine-tuned on the RACE dataset. Inputs are structured by concatenating dialogue context, the question, and a specific option, separated by [SEP] tokens. The framework employs a dual-task training objective:

  1. Correct Answer (CA) Model: Trained using standard cross-entropy loss to identify the correct option among the choices.
  2. Wrong Answer (WA) Model: Trained using binary cross-entropy loss where labels are flipped—wrong options are labeled '1' and the correct option is '0'. This forces the model to learn the specific features that characterize a distractor.

Ensemble Strategy

To make a final prediction, the system combines the scores from both models. Rather than relying solely on the CA score ($P_c$), the ensemble subtracts a weighted version of the WA score ($P_w$): $$\text{Prediction} = \text{argmax}(P_c - w \cdot P_w)$$ The weight ($w$) is optimized via linear regression to balance the contribution of truth-seeking and distractor-avoidance.

Results

The WAE method was evaluated on the DREAM dataset, a dialogue-based MCQA benchmark requiring commonsense reasoning and multi-sentence understanding.

Comparison of accuracy between standard BERT models and the Wrong Answer Ensemble (WAE) variants on the DREAM dataset.

As shown in the results, the WAE framework consistently outperforms baseline fine-tuned BERT models. The BERT-large WAE achieved an accuracy of 69.0%, representing a 2.7% absolute improvement. Notably, the WA model alone ("Wrong" in the table) performed competitively with the standard baseline, indicating that the signal from incorrect answers is highly informative.

Key Insights

  • Learning from Negatives: Explicitly labeling wrong answers allows the model to generalize better and avoid "likely but wrong" options.
  • Model Capacity: BERT-large benefited more from WAE (+2.7%) than BERT-base (+1.7%). This suggests that larger models have the parameter capacity to learn the complex features of distractors without compromising their ability to identify correct answers.
  • Task Sensitivity: The method proved most effective for high-level reasoning tasks. It was less impactful on "matching" questions where the answer is explicitly stated, suggesting that elimination is a strategy best suited for ambiguity.

Conclusion

The Wrong Answer Ensemble demonstrates that modeling what not to choose is as important as modeling what to choose in MCQA. While the current implementation uses a static weighting mechanism, future work could explore dynamic weighting or question-type discriminators to further refine the elimination strategy. Despite the slight increase in computational overhead, the performance gains on reasoning-heavy datasets like DREAM justify the approach.

Summary updated

BibTeX
@inproceedings{kim2020learning,
  author = {Hyeondey Kim and Pascale Fung},
  title = {{Learning to Classify the Wrong Answers for Multiple Choice Question Answering (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7194},
}
Figure