Leveraging Physical and Semantic Features of text item for Difficulty and Response Time Prediction of USMLE Questions

Gummuluri Venkata Ravi Ram, Ashinee Kesanam, Anand Kumar MNAACL 2024

TL;DR

This paper tackles the challenge of automatically predicting the difficulty and response time of medical exam questions using NLP. By leveraging both the textual content and derived numerical features, the authors developed models that achieve promising results, offering a more efficient alternative to traditional, labor-intensive methods for exam development.

What Problem Does This Paper Address?

Standardized exams, like the USMLE, rely on accurate item difficulty and response time predictions to ensure fairness and validity. Traditionally, these metrics are gathered through costly and time-consuming pretesting processes. This paper addresses the need for automated methods to predict these crucial item characteristics directly from the question text itself, thereby reducing costs, improving efficiency, and potentially enhancing test security by minimizing item exposure. The authors aim to advance the state-of-the-art in this area, particularly for high-stakes medical exams where data sharing can be challenging.

Methods

The researchers explored several approaches, starting with a BERT-based regression model as a baseline. They then experimented with different feature extraction and modeling strategies. Key methods include:

  • BERT and BioBERT Embeddings: Utilizing pre-trained language models (BERT and its biomedical variant, BioBERT) to generate rich textual representations of the questions. BioBERT was fine-tuned on question-difficulty pairs to capture domain-specific nuances.
  • Numerical Feature Engineering: Extracting and engineering several numerical features hypothesized to influence difficulty and response time. These include readability scores (using the SMOG index), question length, option count, average option length, and option similarity.
  • Machine Learning Models: Employing a range of regression models, including Random Forest, Gradient Boosting, Linear Regression, Support Vector Regression (SVR), K-Nearest Neighbors (KNN), and Multilayer Perceptron (MLP).
  • Hybrid Approaches: Combining textual embeddings with numerical features for a more comprehensive input to the models. They also explored different ways of concatenating these features.
  • Data Augmentation: To address data imbalance, particularly for higher difficulty levels, they used GPT-3.5 to paraphrase existing questions and generate new instances.

Datasets

The study utilized a dataset comprising 466 previously utilized and retired Multiple Choice Questions (MCQs) from the United States Medical Licensing Examination (USMLE) Steps 1, 2 CK, and 3. This dataset was provided for a Shared Task organized by the ACL Special Interest Group for Building Educational Applications (BEA SIGEDU).

Key Insights and Ideas

  • Textual and Numerical Features are Complementary: The results strongly suggest that both the semantic content of the question (captured by embeddings) and engineered numerical features (like readability and length) are crucial for accurate prediction. Models that combined both generally outperformed those using only one type of feature.
  • Domain-Specific Embeddings Matter: Using BioBERT, a model pre-trained on biomedical text, yielded better results than a general BERT model, highlighting the importance of domain-specific language understanding for medical exams.
  • Deep Learning Models Need Data: The authors observed that while ANNs (a type of deep learning model) showed promise, they struggled with the relatively small dataset size. This led them to shift focus towards more traditional machine learning models, which performed better in this context.
  • Data Augmentation Helps: Paraphrasing questions to balance the dataset, especially for higher difficulty levels, was shown to be beneficial.
  • Hypotheses Validated: The engineered numerical features were derived from hypotheses about factors influencing difficulty and response time (e.g., readability, length). The correlation analysis and model performance generally supported these hypotheses.

Key Results

The paper reports the following RMSE scores on the test data:

  • Difficulty Prediction: Achieved an RMSE of 0.315 with the Random Forest Regressor.
  • Response Time Prediction: Achieved an RMSE of 26.945 with the Multilayer Perceptron.

These results are presented in comparison to other models, with Gradient Boosting, Random Forest, and Multilayer Perceptron generally showing the best performance when using concatenated Bio-BERT embeddings and numerical features.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the limited size of the dataset (466 samples) might have hindered the performance of deep learning models like ANNs. They suggest that deep learning models might require more data to capture patterns effectively.
  • Exploring Deep Learning with More Data: Future work could involve exploring how deep learning models can learn better with limited data or investigating which embeddings are most suitable for such tasks.
  • Identifying More Factors: The authors propose further exploration into other factors that might impact question difficulty and response time, with the goal of incorporating this information into future models to achieve even better RMSE scores.

Summary updated

BibTeX
@inproceedings{ram2024leveraging,
  author = {Gummuluri Venkata Ravi Ram and Ashinee Kesanam and Anand Kumar M},
  title = {{Leveraging Physical and Semantic Features of text item for Difficulty and Response Time Prediction of USMLE Questions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure