Fine-Tuning Encoder-Decoder Models with Contrastive Learning for In-Context Distractor Generation

Elaf Alhazmi, Quan Z. Sheng, Wei Emma Zhang, Mohammed Thanoon, Haojie Zhuang, Behnaz Soltani, Munazza Zaib2025 · DOI

Abstract

Distractor generation is the task of automatically generating plausible yet incorrect options (i.e., distractors) for fill-in-the-blank and multiple-choice questions. In assessment, distractors must be contextually relevant to the given question and answer. Even though recent research works focus on fine-tuning pre-trained encoder-decoder models with data augmentation techniques to generate distractors, these models often fail to capture the full semantic representation of a given question-answer and related distractors. The augmentation methods often rely on expanding the quantity of proposed candidates (i.e., questions or distractors), which can introduce noise into the models without necessarily enhancing their understanding of the deeper semantic relationships between question-answer and related distractors. This paper introduces a novel distractor generation model based on contrastive learning to train the model to recognize essential semantic features necessary to generate in-context distractors. The extensive experiments on two public datasets indicate that contrastive learning introduces a strong baseline model to the distractor generation task. It significantly outperforms recent models, increasing the NDCG@3 score from 24.68 to 32.33 on the MCQ dataset and from 26.66 to 36.68 on the SciQ dataset.

Introduction

In educational technology, the creation of high-quality multiple-choice questions is a complex task. The "distractors" (the incorrect answer choices) must be plausible enough to challenge the learner while remaining clearly incorrect. A common failure mode for existing AI models in Distractor Generation (DG) is the production of options that are related to the correct answer in a general sense but ignore the specific context of the question. For instance, if a question asks for a chemical element related to water ($H_2O$), a standard model might suggest "air" or "gas"—terms related to water but not belonging to the category of chemical elements.

Previous research has attempted to mitigate this by using large-scale data augmentation. However, simply increasing the volume of training data often introduces noise rather than improving the model's grasp of the semantic nuances between a question, its answer, and its distractors.

Methodology

The authors propose a two-stage training pipeline for encoder-decoder models (such as T5 and BART) that integrates contrastive learning to better align semantic representations.

  1. Stage 1: Standard Generation: The model undergoes initial fine-tuning using standard cross-entropy loss to learn the basic sequence-to-sequence (Seq2Seq) task of generating distractor text.
  2. Stage 2: Joint Contrastive Learning: The model is trained with a multi-task objective. It continues to optimize for text generation while simultaneously minimizing a contrastive loss.

The contrastive objective teaches the model to map a Question-Answer pair (the anchor) and its ground-truth distractors (positives) close together in a high-dimensional embedding space. Conversely, distractors from other questions in the same batch (negatives) are pushed further away. The study explored two specific loss functions: InfoNCE, which utilizes multiple negative examples per batch to sharpen semantic focus, and Triplet Loss, which maintains a specific margin between an anchor, a positive, and a single negative.

The proposed contrastive learning framework, showing the integration of InfoNCE and Triplet losses within the encoder-decoder architecture to align question-answer pairs with relevant distractors.

Experimental Results

The models were evaluated on two benchmark datasets: SciQ (13,700 science questions) and MCQ/Dgen (2,580 general knowledge questions). The results demonstrate that contrastive learning provides a significant performance boost over standard fine-tuning and other state-of-the-art methods.

Main experimental results comparing the contrastive learning approach against various baselines across MCQ and SciQ datasets.

Key findings include:

  • Performance Gains: On the MCQ dataset, the NDCG@3 score improved from 24.68 to 32.33. On the SciQ dataset, it rose from 26.66 to 36.68.
  • Semantic Alignment: Contrastive learning proved more effective than simply increasing data volume, as it forces the model to capture fine-grained semantic distinctions.
  • Comparison with Large Models: The specialized T5 model with contrastive tuning significantly outperformed GPT-3 (few-shot), highlighting the value of task-specific optimization over general-purpose scale.
  • Human Evaluation: In blind tests, human graders rated the distractors from the contrastive T5 model higher in relevance, difficulty, and fluency than the original ground-truth distractors in the datasets.

Discussion and Conclusion

While the contrastive approach yields substantial improvements, challenges remain. Current evaluation metrics like NDCG and F1 rely on exact token matching, which may penalize semantically valid distractors that do not appear in the ground truth. Additionally, generative models can still produce repetitive outputs. Future work will explore integrating contrastive learning into candidate selection frameworks to further refine the quality of generated distractors.

Summary updated

Abstract

Distractor generation is the task of automatically generating plausible yet incorrect options (i.e., distractors) for fill-in-the-blank and multiple-choice questions.In assessment, distractors must be contextually relevant to the given question and answer.Even though recent research works focus on fine-tuning pre-trained encoder-decoder models with data augmentation techniques to generate distractors, these models often fail to capture the full semantic representation of a given question-answer and related distractors.The augmentation methods often rely on expanding the quantity of proposed candidates (i.e., questions or distractors), which can introduce noise into the models without necessarily enhancing their understanding of the deeper semantic relationships between question-answer and related distractors.This paper introduces a novel distractor generation model based on contrastive learning to train the model to recognize essential semantic features necessary to generate in-context distractors.The extensive experiments on two public datasets indicate that contrastive learning introduces a strong baseline model to the distractor generation task.It significantly outperforms recent models, increasing the NDCG@3 score from 24.68 to 32.33 on the MCQ dataset and from 26.66 to 36.68 on the SciQ dataset.
BibTeX
@misc{alhazmi2025fine,
  author = {Elaf Alhazmi and Quan Z. Sheng and Wei Emma Zhang and Mohammed Thanoon and Haojie Zhuang and Behnaz Soltani and Munazza Zaib},
  title = {{Fine-Tuning Encoder-Decoder Models with Contrastive Learning for In-Context Distractor Generation}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.533},
}
Figure