TL;DR
This paper explores using Large Language Models (LLMs) to predict the difficulty of medical multiple-choice questions (MCQs). Instead of directly asking the LLM about difficulty, they simulate test-takers by having the LLM answer questions under various simulated conditions, revealing that item difficulty is more about the test-taker population than the item itself.
What Problem Does This Paper Address?
Accurately assessing the difficulty of exam questions is crucial for creating effective assessments that can reliably gauge a person's knowledge. Traditionally, this has been a manual, expert-driven process, which is time-consuming and prone to inconsistencies. While Natural Language Processing (NLP) has enabled automated prediction of item difficulty, existing methods often focus on linguistic features of the questions themselves. This paper questions that assumption, proposing that the difficulty of an item might be more influenced by the characteristics of the population answering it, rather than just the item's content. They aim to address this by simulating diverse test-taker behaviors using LLMs to predict MCQ difficulty.
Methods
The core of this work is leveraging LLMs (specifically, ChatGPT-3.5) to simulate test-taker responses. They developed three main strategies for generating features:
- LLM Answering Questions (Prompt #1): The LLM is asked to answer MCQs and provide a justification. Features are extracted from its response, including whether it answered correctly, the length of its justification, and basic item metadata.
- Decomposing into Yes/No Questions (Prompt #2): Each MCQ option is turned into a yes/no question. The LLM answers these, and features are derived from its responses to these sub-items, aiming for more granular insights into difficulty.
- Mutilated Stems (Prompt #3): The LLM's input (the question stem) is progressively "mutilated" by removing content words. The assumption is that harder questions will be unanswerable even with minor mutilation, while easier ones will remain answerable. Features capture how well the LLM performs as the stem is degraded.
- Modified "Temperatures": The "temperature" parameter of the LLM, which controls randomness/creativity, is varied. This simulates test-takers with different levels of objectivity. Features are derived from how the LLM's performance changes with temperature.
These extracted features are then fed into a Ridge Regression model to predict the ground truth item difficulty.
Datasets
The study uses a dataset of 667 MCQs from the USMLE Steps 1, 2 CK, and 3. This dataset was split into 467 training questions and 200 test questions. The dataset includes item stems, answer options, correct answers, and metadata like item type, exam step, item difficulty (ground truth), and time intensity.
Key Insights and Ideas
- Difficulty is in the Eye of the Beholder (Simulated): The most significant finding is that simulating test-taker behavior, rather than just analyzing item content, yields strong predictors of difficulty. This supports the hypothesis that population characteristics are key.
- Mutilation and Temperature as Simulators: The "mutilated stem" and "temperature variation" strategies proved particularly effective. They successfully simulate diverse test-taker behaviors (e.g., skimming vs. detailed reading, or varying levels of objectivity/emotional influence) using a single LLM.
- LLM as a Proxy Test-Taker: The LLM, when prompted appropriately, can effectively simulate human test-taking behavior, providing a scalable way to generate data for difficulty prediction.
- Justification Length Isn't Everything: Surprisingly, the length of the LLM's justification for its answers (from Prompt #1) was not a strong predictor of difficulty, contrary to initial expectations. However, justifications from the Yes/No sub-item strategy did show promise.
Key Results
- The system achieved the second-best rank in the BEA 2024 Shared Task on Automated Prediction of Item Difficulty, with a Root-Mean Squared Error (RMSE) of 0.303.
- The
INCORRECTfeature (whether the LLM answered correctly in the standard prompt) emerged as the most significant predictor from Prompt #1. - Features derived from the mutilated stem strategy (e.g.,
FIRST_MUT_INCORRECT,LAST_MUT_INCORRECT) showed strong negative correlations, indicating that if an LLM can answer a heavily mutilated question, it's likely an easy item. - Features from the temperature variation strategy also performed well, with
FIRST_TEMP_INCORRECTshowing a negative correlation, suggesting that if an LLM can answer correctly even at high temperatures (less objective), the item is likely easy. - The best predictive model was achieved using a Ridge Regression with a regularization parameter
aset to 756.
Limitations and Future Work
- Challenging Task: Despite competitive results, the task remains challenging, with even the best systems barely outperforming a simple baseline.
- LLM Refusal: While not encountered in this specific experiment, the paper notes that LLMs might refuse to answer questions, which could be an indicator of difficulty.
- Future Work: The authors suggest exploring combinations of the successful mutilation and temperature strategies, as well as other similar simulation approaches, to further improve predictive models. They also highlight the need for more robust evaluation metrics that can better differentiate between feature performances.
Summary updated