Distractor Generation for Multiple Choice Questions Using Learning to Rank

Chen Liang, Xiao Yang, Neisarg Dave, Drew Wham, Bart Pursel, C. Lee GilesNAACL 2018

TL;DR

This paper tackles the challenge of automatically generating plausible distractors (incorrect answer choices) for multiple-choice questions (MCQs). They propose using "learning to rank" models, which are trained on existing MCQs to learn what makes a good distractor, outperforming traditional similarity-based methods and neural network approaches.

What Problem Does This Paper Address?

Creating effective multiple-choice questions (MCQs) is a crucial but time-consuming task for educators. While the question stem and correct answer (key) are relatively straightforward, generating good distractors—plausible but incorrect options that genuinely test a student's understanding—is particularly challenging. Existing methods often rely on simple similarity measures (like WordNet or word embeddings) or heuristic rules, which can lead to distractors that are too obvious, grammatically incorrect, or semantically unrelated. This paper aims to automate the generation of high-quality distractors by learning from real-world exam questions, moving beyond simple similarity metrics.

Methods

The core contribution of this paper is framing distractor generation (DG) as a learning to rank problem. Instead of directly generating distractors, they train models to rank a set of candidate distractors, prioritizing those that are most likely to be good. They explore two main types of models:

  • Feature-based Models: These models transform a question stem, key, and candidate distractor into a rich feature vector. This vector includes a wide array of features, such as:
  • Semantic Similarity: Using GloVe embeddings to capture meaning.
  • Syntactic and Lexical Features: Part-of-speech tag similarity, edit distance (for spelling), token overlap, and length comparisons.
  • Domain-Specific Features: Word frequency (as a proxy for difficulty) and singular/plural consistency.
  • Wikipedia Embeddings: For entity-based similarity. They then train standard machine learning classifiers like Logistic Regression, Random Forest, and LambdaMART (a gradient-boosted ranking model) on these features.

  • NN-based Models: Inspired by Generative Adversarial Networks (GANs), they propose an adversarial framework. A generator tries to create distractors, while a discriminator tries to distinguish real distractors from generated ones. They also incorporate a Decomposable Attention model to better capture semantic similarities between the question and distractors.

A novel aspect is their Cascaded Learning Framework. This involves a two-stage process: a first-stage ranker quickly filters down a large set of candidate distractors to a manageable subset, and a second-stage ranker then applies more complex models and features to the reduced set. This significantly speeds up the process and allows for more sophisticated analysis.

Datasets

The authors introduce two datasets for evaluating distractor generation models:

  • SciQ: A recently released dataset of 13.7K multiple-choice science questions crowdsourced from the web, covering various science disciplines.
  • MCQL: A new dataset of 7.1K MCQs also crawled from the web, focusing on biology, physics, and chemistry.

These datasets are made available as benchmarks for future research in DG.

Key Insights and Ideas

  • Framing DG as Ranking is Powerful: Treating distractor generation as a ranking problem, where the goal is to order potential distractors by their quality, proves more effective than directly generating them or relying solely on simple similarity measures.
  • Rich Features Trump Simple Similarity: The feature-based models, which combine a wide range of semantic, syntactic, and lexical features, significantly outperform the NN-based model that relies primarily on word embeddings. This highlights the importance of diverse linguistic cues for generating good distractors.
  • Ensemble Methods Excel: Gradient-boosted tree models like LambdaMART and Random Forests, which are ensemble methods, demonstrate superior performance, suggesting they are adept at capturing complex relationships between features and distractor quality.
  • Cascading Improves Efficiency and Effectiveness: The two-stage cascaded learning approach effectively reduces the search space for distractors, making the overall process more efficient and allowing the second stage to focus on more challenging examples.
  • NNs Struggle with Limited Data: The NN-based model underperforms, likely due to the limited size of the training data and its sole reliance on word embeddings, missing crucial information captured by other features.

Key Results

  • Feature-based ensemble models (Random Forest and LambdaMART) outperform both the NN-based method and unsupervised baselines. For example, on the SciQ dataset, LambdaMART achieved an MRR of 49.3%, significantly higher than the NN model's 25.7%.
  • The cascaded learning approach is effective. The paper shows that using a two-stage ranker leads to better performance compared to a single-stage approach, even when the single-stage model uses a comparable number of negative samples.
  • Top features are consistent across datasets. Embedding similarity between the answer and distractor is consistently the most important feature, followed by string similarities like Token Sim, ED, and Suffix, especially in the MCQL dataset.

Limitations and Future Work

  • NN Model Performance: The authors acknowledge that their NN-based model performed worse than feature-based methods, attributing this to limited training data and the model's reliance on word embeddings.
  • Future Work: The authors suggest designing a user interface to help educators with DG and collecting more user data to improve model training. They also mention the potential for further research into user interfaces and model training with more data.

Summary updated

BibTeX
@inproceedings{liang2018distractor,
  author = {Chen Liang and Xiao Yang and Neisarg Dave and Drew Wham and Bart Pursel and C. Lee Giles},
  title = {{Distractor Generation for Multiple Choice Questions Using Learning to Rank}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure