Abstract
Distractor generation is the task of automatically generating plausible yet incorrect options (i.e., distractors) for fill-in-the-blank and multiple-choice questions. In assessment, distractors must be contextually relevant to the given question and answer. Even though recent research works focus on fine-tuning pre-trained encoder-decoder models with data augmentation techniques to generate distractors, these models often fail to capture the full semantic representation of a given question-answer and related distractors. The augmentation methods often rely on expanding the quantity of proposed candidates (i.e., questions or distractors), which can introduce noise into the models without necessarily enhancing their understanding of the deeper semantic relationships between question-answer and related distractors. This paper introduces a novel distractor generation model based on contrastive learning to train the model to recognize essential semantic features necessary to generate in-context distractors. The extensive experiments on two public datasets indicate that contrastive learning introduces a strong baseline model to the distractor generation task. It significantly outperforms recent models, increasing the NDCG@3 score from 24.68 to 32.33 on the MCQ dataset and from 26.66 to 36.68 on the SciQ dataset.
Introduction
In educational technology, the creation of high-quality multiple-choice questions is a complex task. The "distractors" (the incorrect answer choices) must be plausible enough to challenge the learner while remaining clearly incorrect. A common failure mode for existing AI models in Distractor Generation (DG) is the production of options that are related to the correct answer in a general sense but ignore the specific context of the question. For instance, if a question asks for a chemical element related to water ($H_2O$), a standard model might suggest "air" or "gas"—terms related to water but not belonging to the category of chemical elements.
Previous research has attempted to mitigate this by using large-scale data augmentation. However, simply increasing the volume of training data often introduces noise rather than improving the model's grasp of the semantic nuances between a question, its answer, and its distractors.
Methodology
The authors propose a two-stage training pipeline for encoder-decoder models (such as T5 and BART) that integrates contrastive learning to better align semantic representations.
- Stage 1: Standard Generation: The model undergoes initial fine-tuning using standard cross-entropy loss to learn the basic sequence-to-sequence (Seq2Seq) task of generating distractor text.
- Stage 2: Joint Contrastive Learning: The model is trained with a multi-task objective. It continues to optimize for text generation while simultaneously minimizing a contrastive loss.
The contrastive objective teaches the model to map a Question-Answer pair (the anchor) and its ground-truth distractors (positives) close together in a high-dimensional embedding space. Conversely, distractors from other questions in the same batch (negatives) are pushed further away. The study explored two specific loss functions: InfoNCE, which utilizes multiple negative examples per batch to sharpen semantic focus, and Triplet Loss, which maintains a specific margin between an anchor, a positive, and a single negative.
Experimental Results
The models were evaluated on two benchmark datasets: SciQ (13,700 science questions) and MCQ/Dgen (2,580 general knowledge questions). The results demonstrate that contrastive learning provides a significant performance boost over standard fine-tuning and other state-of-the-art methods.
Key findings include:
- Performance Gains: On the MCQ dataset, the NDCG@3 score improved from 24.68 to 32.33. On the SciQ dataset, it rose from 26.66 to 36.68.
- Semantic Alignment: Contrastive learning proved more effective than simply increasing data volume, as it forces the model to capture fine-grained semantic distinctions.
- Comparison with Large Models: The specialized T5 model with contrastive tuning significantly outperformed GPT-3 (few-shot), highlighting the value of task-specific optimization over general-purpose scale.
- Human Evaluation: In blind tests, human graders rated the distractors from the contrastive T5 model higher in relevance, difficulty, and fluency than the original ground-truth distractors in the datasets.
Discussion and Conclusion
While the contrastive approach yields substantial improvements, challenges remain. Current evaluation metrics like NDCG and F1 rely on exact token matching, which may penalize semantically valid distractors that do not appear in the ground truth. Additionally, generative models can still produce repetitive outputs. Future work will explore integrating contrastive learning into candidate selection frameworks to further refine the quality of generated distractors.
Summary updated