TL;DR
This paper explores using transfer learning to predict the difficulty (P-value) and response time of multiple-choice medical exam questions. They found that transfer learning can indeed improve predictions, particularly for response time, and identified that the question stem is most informative for difficulty, while the entire item is crucial for response time.
What Problem Does This Paper Address?
Standardized exams rely on questions with well-defined difficulty and response time characteristics to ensure fairness and efficiency. Traditionally, these parameters are determined by embedding new questions as unscored items in live exams, a process that is time-consuming and costly. This paper addresses the need for a more efficient method to predict these item parameters before they are used in live exams. The authors propose leveraging transfer learning, a technique where knowledge gained from one task is applied to a related task, to improve the prediction accuracy of item difficulty and response time, especially in scenarios where data for one parameter might be abundant while data for the other is scarce.
Methods
The core of this research lies in applying transfer learning to predict item parameters. They used ELMo (Embeddings from Language Models) to generate contextualized word embeddings for different parts of the multiple-choice questions (MCQs): the item stem, the answer options, or the entire item. These embeddings were then fed into a Bidirectional LSTM network to extract higher-level "encoding features."
Two transfer learning approaches were investigated:
- Method 1 (Auxiliary Task): The ELMo model was initialized with pre-trained parameters and then trained on the MCQ data to predict either P-value or Response Time. The target variable for training was the same as the target variable for prediction.
- Method 2 (Transfer Learning): Similar to Method 1, but the ELMo parameters were updated during training on an auxiliary task. For instance, if the goal was to predict P-value, the model was trained on Response Time data as an auxiliary task, and vice-versa.
Linear regression was used to predict the item parameters from the extracted embeddings and encodings. They compared these methods against a ZeroR baseline (predicting the majority class) and against each other.
Datasets
The study utilized a dataset of approximately 18,000 multiple-choice questions (MCQs) from a high-stakes medical licensing exam (USMLE®). These questions were administered as unscored pretest items over six annual cycles between 2010 and 2015. The dataset included P-values (proportion of correct answers) and average response times for each item.
Key Insights and Ideas
- Transfer Learning Pays Off (Sometimes): Transfer learning, particularly when using Response Time to aid P-value prediction, showed significant improvements. This suggests that knowledge about how long an item takes to answer can indeed inform its difficulty.
- Asymmetry in Transfer: The benefit of transfer learning was not symmetrical. While using Response Time as an auxiliary task improved P-value prediction, using P-value to predict Response Time did not yield similar gains.
- Stem for Difficulty, Whole Item for Speed: The research highlights a crucial difference in what parts of an item are informative. The item stem (the clinical case description) was most predictive of item difficulty (P-value). Conversely, predicting response time benefited most from considering the entire item, including the stem and all answer options.
- Response Time Data is "Easier" to Model: The authors hypothesize that the normally distributed nature of Response Time, compared to the potentially skewed distribution of P-values, might make it a more amenable target for machine learning models, thus facilitating better transfer.
Key Results
- Both transfer learning methods (Method 1 and Method 2) achieved statistically significant improvements in predicting both P-value and Response Time compared to the ZeroR baseline, with RMSE decreases.
- Method 2 (where ELMo parameters were updated) significantly improved the prediction of Response Time when P-value was used as an auxiliary task.
- For P-value prediction, the stem alone provided the best results in Method 1.
- For Response Time prediction, the full item outperformed other configurations in Method 1.
- The best P-value prediction using Method 2 was achieved when using the stem and answer options together, with an RMSE of 23.04.
- The best Response Time prediction using Method 2 was achieved when using the stem and answer options together, with an RMSE of 0.33.
Limitations and Future Work
- P-value Skewness: The authors acknowledge that the skewed distribution of P-values might hinder learning compared to the more normally distributed Response Time. They suggest future work could explore normalizing these distributions.
- Limited Transfer Benefit: The lack of significant improvement when using P-value to predict Response Time suggests that further research is needed to understand this asymmetry.
- Practical Implications: While the study demonstrates the potential of transfer learning, further work is needed to fully integrate these methods into practical item development workflows. Future research could also explore other NLP techniques and more sophisticated transfer learning strategies.
Summary updated