TL;DR
This paper explores using Large Language Models (LLMs) to predict the difficulty and response time of medical exam questions. By augmenting existing datasets with LLM-generated answers and employing transformer-based models, the researchers found that LLMs can significantly improve prediction accuracy, particularly for question difficulty, a task previously considered more challenging.
What Problem Does This Paper Address?
Creating high-stakes medical licensing exams, like the USMLE, requires meticulously crafted questions that accurately assess a candidate's knowledge. Traditionally, determining a question's difficulty and the average time it takes to answer (response time) relied on pretesting. This involves embedding new questions alongside scored ones in live exams. However, this method is resource-intensive and impractical due to the sheer volume of questions needed. The BEA 2024 Shared Task aims to address this by developing automated methods for predicting item difficulty and response time using NLP techniques, making exam creation more efficient, secure, and informative.
Methods
The core of this research lies in leveraging Large Language Models (LLMs) for data augmentation and prediction. The authors used three zero-shot LLMs (Falcon-7B, Meditron-7B, and Mistral-7B) to generate answers for the USMLE questions. This augmented dataset was then preprocessed, including scaling target labels, creating new feature combinations, and cleaning text.
Two main types of transformer-based models were fine-tuned on this augmented data:
- Fine-tuned BERT: A BERT model was used to generate contextualized text representations, followed by a regression head to predict difficulty and response time.
- Fine-tuned GPT-2: Similarly, a GPT-2 model was fine-tuned for the regression tasks.
Additionally, the researchers explored a linear probing approach using v-Support Vector Regression (v-SVR) on frozen BERT embeddings. This method was investigated to mitigate overfitting, a common issue with transformer models on small datasets.
Datasets
The primary dataset used in this study is the one provided for the BEA 2024 Shared Task, consisting of retired USMLE Multiple-Choice Questions (MCQs). This dataset was divided into an initial training set of 466 samples and a test set of 201 samples. The authors augmented this dataset by generating LLM responses, creating a larger, richer dataset for their experiments.
Key Insights and Ideas
- Difficulty is Harder: Predicting question difficulty proved to be a significantly more challenging task than predicting response time.
- Question Text is Crucial: The most successful models consistently incorporated the original question text, highlighting its importance for both prediction tasks.
- LLMs Boost Performance: LLM-generated answers had a positive impact on prediction accuracy, especially for the more challenging difficulty prediction task.
- Linear Probing for Generalization: Training a v-SVR model on frozen pre-trained transformer features (linear probing) demonstrated better generalization capacity than end-to-end fine-tuning, likely due to the limited size of the training dataset.
- AnswerKey + LLM Synergy: The AnswerKey feature became particularly useful when combined with LLM answers, improving the performance of v-SVR+BERT models.
Key Results
The paper reports results from both cross-validation and a final test set evaluation.
Response Time Prediction: * The best MSE was achieved by fine-tuned BERT models using "q_llms_a" features (MSE: 0.0331). * For Kendall T correlation, fine-tuned GPT-2 with "q_answers" features performed best (0.4767). * Post-competition v-SVR+BERT models achieved impressive results, with "q_llms_a_key" features yielding the lowest MSE (0.0582) and "q_llms_a" features closely following (0.0638).
Difficulty Prediction: * The best MSE was achieved by fine-tuned BERT models using "q_llms_a" features (MSE: 0.0500). * The highest Kendall T correlation was obtained by SVR+BERT with "q_llms_a_key" features (0.1592). * The post-competition SVR+BERT models showed consistent MSE and Kendall T values, suggesting effective overfitting mitigation.
Limitations and Future Work
The authors acknowledge several limitations:
- LLM Size: Due to computational constraints, they could only prompt smaller versions (7B parameters) of LLMs. Using larger models might yield better results.
- Dataset Size: The limited number of samples made transformer models prone to overfitting.
- Overfitting: The final results indicated some level of overfitting in their models.
Future work could focus on:
- Exploring methods to avoid overfitting, such as dropout, frozen layers, and regularization terms.
- Investigating larger LLMs for answer generation.
- Further exploring the synergy between LLM answers and traditional features like AnswerKey.
Summary updated