Item Difficulty and Response Time Prediction with Large Language Models: An Empirical Analysis of USMLE Items

Okan Bulut, Guher Gorgun, Bin TanNAACL 2024

TL;DR

This paper explores using large language models (LLMs) to predict the difficulty and response time of medical licensing exam questions. They found that while LLMs like BiomedBERT show promise for predicting item difficulty, predicting response time is more straightforward and benefits from simpler models like FastText.

What Problem Does This Paper Address?

Standardized tests, like the USMLE, rely on understanding how difficult individual questions are and how long they take to answer. Traditionally, this information comes from pretesting questions on large groups of test-takers, which is costly and time-consuming. Researchers have been looking for ways to predict these "item characteristics" using only the text of the questions themselves. Previous attempts using expert opinions or simpler text analysis methods have had mixed success. This paper tackles the challenge of using advanced NLP techniques, specifically LLMs, to automate this prediction process for high-stakes medical exams.

Methods

The authors employed a multi-pronged approach to extract features from the USMLE questions:

  • Linguistic Features: They used Coh-Metrix, a tool that analyzes text at multiple linguistic and discourse levels (e.g., sentence complexity, cohesion, narrativity), to extract a rich set of features.
  • LLM Embeddings: They leveraged state-of-the-art LLMs:
  • BiomedBERT: A BERT model pre-trained on biomedical literature, used to generate sentence embeddings for item stems and response options. Cosine similarity was then calculated between these embeddings to capture semantic relationships.
  • FastText: A word embedding technique that considers word n-grams, used to generate vector representations for words. These were concatenated to create feature vectors for each response option.
  • Metadata: They also included item metadata, such as item type (text-only vs. image-based) and the specific USMLE step the item belonged to.
  • Model Training: For item difficulty prediction, they used Principal Component Analysis (PCA) for dimension reduction and Lasso regression for feature selection and regularization. For response time prediction, they also used feature selection and dimension reduction, followed by Lasso regression.

Datasets

The study utilized a dataset of 667 retired multiple-choice items from the United States Medical Licensing Examination® (USMLE®). The dataset was released in two stages: an initial set of 446 items for feature extraction and model building, and a subsequent set of 201 items for testing. The dataset included item stems, response options, and metadata such as item difficulty and average response time.

Key Insights and Ideas

  • LLMs are powerful, but not a silver bullet: While BiomedBERT showed strong performance in predicting item difficulty, it wasn't a universally superior solution for all aspects.
  • Response time prediction is more tractable: Predicting how long a question will take to answer appears to be a more straightforward task using linguistic features compared to predicting difficulty. This is likely because reading time is more directly correlated with text complexity.
  • Linguistic features alone might be insufficient for difficulty: The results suggest that purely linguistic features might not fully capture the complex interplay of factors that determine item difficulty.
  • Domain-specific LLMs matter: Using BiomedBERT, a model pre-trained on biomedical text, likely provided an advantage in understanding the nuances of medical exam questions.

Key Results

  • Item Difficulty: The best model for predicting item difficulty, using BiomedBERT embeddings and other features, achieved a Mean Squared Error (MSE) of .064 and a Root Mean Squared Error (RMSE) of .253. This ranked 20th out of 43 submissions in the competition.
  • Response Time: The best model for predicting response time, primarily using FastText embeddings and some metadata, achieved an RMSE of 31.48 and an MAE of 23.54. This model performed slightly better than a baseline dummy regressor and ranked 24th out of 34 submissions.

Limitations and Future Work

The authors acknowledge that predicting item difficulty using only linguistic features remains challenging. They suggest that factors beyond textual content might be necessary to achieve higher accuracy. Future work could explore incorporating these additional factors. The limited size of the dataset (N=466 for training) also presented challenges, necessitating dimension reduction techniques.

Summary updated

BibTeX
@inproceedings{bulut2024item,
  author = {Okan Bulut and Guher Gorgun and Bin Tan},
  title = {{Item Difficulty and Response Time Prediction with Large Language Models: An Empirical Analysis of USMLE Items}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure