TL;DR
This paper introduces a novel, fully automatic method for generating plausible distractors (incorrect answer options) for multiple-choice questions, particularly for high-stakes medical exams. By combining concept embeddings with information retrieval, the system can suggest distractors that are remarkably similar to those created by human experts, significantly aiding in the creation of robust and challenging assessments.
What Problem Does This Paper Address?
Creating high-quality multiple-choice questions (MCQs) is a cornerstone of educational assessment, especially in demanding fields like medicine. However, a significant bottleneck in this process is the generation of effective distractors. These incorrect options are crucial for controlling test difficulty, preventing random guessing, and accurately differentiating between students' knowledge levels. Human item-writers find this task challenging and time-consuming, leading to a substantial portion of distractors being ineffective or rarely chosen. The sheer scale of item banks required for standardized exams (e.g., USMLE) translates into enormous development costs, creating a strong incentive for automation. This paper aims to automate the generation of plausible distractors, addressing the gap in efficient and effective MCQ item development.
Methods
The core of the proposed method lies in a two-stage approach:
- Concept Embeddings for Candidate Generation: The system leverages pre-trained embedding vectors from lexical-semantic databases like the Unified Medical Language System (UMLS) and Freebase. These embeddings represent concepts as vectors, allowing for the quantification of semantic similarity. Given a question stem and its correct answer, the system identifies candidate distractors by finding concepts that are semantically similar to both the stem and the correct answer. This initial step generates a list of potential distractors.
- Information Retrieval for Re-ranking: To refine the initial list and prioritize more plausible distractors, the system employs an information retrieval (IR) re-ranking strategy. The question stem is used as a query to search a large text collection (specifically, 2013 MEDLINE citations). The premise is that distractors that co-occur with the stem in relevant documents are more likely to be plausible. Candidates are then re-ranked based on the order in which they appear in documents retrieved by the stem query.
Datasets
The study utilizes two datasets of multiple-choice questions from the United States Medical Licensing Examination (USMLE):
- Private Data Set: 1,441 MCQs administered or pretested during the 2008 USMLE administration. These questions are not publicly available due to test security reasons.
- Public Data Set: 369 publicly available MCQs, comprising questions from the USMLE Step 1 2015 and 2016 sample booklets, and the USMLE Step 2 2017 sample booklet.
These datasets provide the question stem, all answer options, and information on which option is the correct answer.
Key Insights and Ideas
- Semantic Similarity is Key: The fundamental idea is that effective distractors are semantically similar to the correct answer and the question stem. Concept embeddings provide a powerful way to quantify this similarity.
- IR Boosts Plausibility: Simply relying on semantic similarity isn't enough. Re-ranking candidates using information retrieval, based on how often they co-occur with the stem in real-world text, significantly improves the likelihood of suggesting distractors that human item-writers would actually use.
- UMLS Outperforms Freebase for Medical Domain: While Freebase has a larger number of concepts, UMLS proved to be a more effective resource for medical domain distractors due to its specialized coverage.
- Broader Semantic Relations are Better: Using broader semantic relations (like "same semantic type" - STY) for candidate generation, rather than very narrow ones (like "sibling" - SIB), leads to better performance, especially when considering a larger number of top suggestions.
- Diminishing Returns: The effectiveness of suggesting more distractors plateaus. The study found that the return diminishes when suggesting more than 20 distractors.
Key Results
The proposed method, combining concept embeddings and IR re-ranking, significantly outperforms a baseline of random distractor prediction.
- The system can match approximately 1 in 5 human-produced distractors when suggesting the top 20 candidates.
- The IR re-ranking approach showed a notable improvement over using concept embeddings alone, with improvements ranging from 2% to 53% depending on the number of top suggestions considered.
- For the top 20 suggestions, the STY relation (same semantic type) outperformed the SIB relation (sibling) by a considerable margin, yielding 142 hits versus 82 hits for the public dataset and 572 hits versus 382 hits for the private dataset.
- The recall rate for the top 20 suggestions using the STY relation was 0.20 for the private dataset and 0.16 for the public dataset.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Assumption of Human-Generated Distractor Quality: The evaluation assumes that human-produced distractors are always the "best." However, human item-writers also struggle with this task, meaning some automatically generated distractors might be perfectly suitable even if not used in the original items.
- Conservative Evaluation: The current evaluation might be too conservative, potentially underestimating the quality of generated distractors.
- Future Evaluation with Human Item-Writers: A planned future evaluation involves presenting human item-writers with lists of automatically generated distractors to assess their suitability.
- Item Response Theory (IRT) Assessment: A longer-term evaluation could involve using IRT to assess the quality of distractors by collecting data from examinees.
- Stem Control: The system does not control the question stem, meaning there's a possibility that a generated distractor could inadvertently become an alternative correct answer. This is partially mitigated by ensuring synonyms of the correct answer are not suggested and by the final human selection process.
- Exploring Other Ontological Relations: Investigating other semantic relations within UMLS (e.g., "RO" - other relation) could further enhance performance.
- Alternative Querying Strategies: Breaking down the stem into smaller components for IR querying might yield better results.
- Machine Learning Models: Exploring other machine learning models could further enhance the number of prediction hits.
Summary updated