Using Machine Learning to Predict Item Difficulty and Response Time in Medical Tests

Mehrdad Yousefpoori-Naeim, Shayan Zargari, Zahra HatamiNAACL 2024

TL;DR

This paper explores using machine learning to predict the difficulty and response time of medical test items before they are administered, saving time and cost in test development. They found that while word embeddings offered limited improvement, linguistic features related to text cohesion were strong predictors of difficulty, and simple length-based features dominated response time prediction.

What Problem Does This Paper Address?

Developing high-stakes medical tests, like the USMLE, requires careful selection of items to ensure equivalent test versions and accurate difficulty and time estimations. Traditionally, this is done through pretesting, which is expensive and raises security concerns. The authors aim to address this by developing machine learning models that can predict item difficulty and response time directly from the item text, bypassing the need for pretesting. This would significantly streamline the test development process and reduce associated costs.

Methods

The researchers employed a variety of machine learning algorithms, including Linear Regression, Ridge Regression, Lasso Regression, ElasticNet, SGD, SVR, Decision Tree, Random Forest, Gradient Boosting, Extra Trees, AdaBoost, K-Neighbors, MLP, XGBoost, and CatBoost. They adopted a three-model approach:

  1. Model 1: Utilized a set of linguistic and non-linguistic features, excluding word embeddings.
  2. Model 2: Incorporated MPNet word embeddings (reduced to 15 components via PCA) alongside the features from Model 1.
  3. Model 3: Employed an ensemble method to combine the strengths of various algorithms.

Feature selection and reduction techniques were used to mitigate multicollinearity and prevent overfitting. The models were evaluated using Root Mean Square Error (RMSE) on both a test set and through 5-fold cross-validation.

Datasets

The study used a retired sample of 667 multiple-choice questions from the United States Medical Licensing Examination (USMLE). This dataset was part of a shared task at the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).

Key Insights and Ideas

  • Embeddings' Limited Impact: Despite the promise of advanced language models, MPNet embeddings provided only a marginal improvement in prediction accuracy when added to models already rich in other features. This suggests that for this specific task and dataset, the information captured by embeddings was largely redundant or not sufficiently discriminative compared to other features.
  • Cohesion is Key for Difficulty: Features measuring text cohesion, particularly those from the Coh-Metrix tool, emerged as the most significant predictors of item difficulty. This highlights that how well the ideas and arguments in an item are connected is crucial for how difficult test-takers perceive it to be.
  • Length Rules Response Time: For predicting response time, simple length-based features, such as stem length and maximum sentence length, were the dominant factors. This aligns with intuition: longer texts generally take more time to read and process.
  • Ensemble Method Ineffectiveness: The ensemble approach (Model 3) did not yield performance improvements. This was attributed to a wide gap in performance between the top-performing individual algorithms and the rest, suggesting that combining them did not offer a synergistic benefit.

Key Results

  • RMSE Performance: Models 1 and 2 showed very similar performance, with Model 2 (including embeddings) being only marginally better. Model 3 (ensemble) did not improve upon Model 2. For difficulty prediction, the best test RMSE was 0.269 (Model 2 and 3), and for response time, it was 23.271 (Model 2 and 3).
  • Feature Importance (Difficulty): Coh-Metrix features like PCTEMPz, LSASSpd, and Rare Words Sum were among the top predictors for difficulty.
  • Feature Importance (Response Time): Sentence Length Maximum, Stem Length, and Medical Terms Sum were the top predictors for response time.

Limitations and Future Work

  • Feature Quality Dependency: The accuracy of extracted features is highly dependent on the quality of stem text preprocessing. Medical items, with their embedded data (e.g., lab results), can pose unique challenges to standard NLP preprocessing, potentially impacting feature calculations and coherence measures.
  • Algorithm-Specific Feature Importance: Feature importance scores can vary significantly between different ML algorithms, even if their overall performance (RMSE) is similar. The authors suggest experimental studies with feature manipulation for a more reliable understanding of individual feature contributions.
  • Future Work - Item Writer Data: The authors propose collecting data directly from item writers, such as their difficulty ratings during item creation, as a potentially valuable and cost-effective feature for future models. This could offer a unique perspective on perceived difficulty.

Summary updated

BibTeX
@inproceedings{yousefpoorinaeim2024using,
  author = {Mehrdad Yousefpoori-Naeim and Shayan Zargari and Zahra Hatami},
  title = {{Using Machine Learning to Predict Item Difficulty and Response Time in Medical Tests}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure