ITEC at BEA 2024 Shared Task: Predicting Difficulty and Response Time of Medical Exam Questions with Statistical, Machine Learning, and Language Models

Anaïs Tack, Siem Buseyne, Changsheng Chen, Robbe D’hondt, Michiel De Vrindt, Alireza Gharahighehi, Sameh Metwaly, Felipe Kenji Nakano, Ann-Sophie NoreillieNAACL 2024

TL;DR

This paper tackles the challenge of automatically predicting the difficulty and response time of medical exam questions, a crucial task for ensuring fair and effective assessments. The authors found that simpler models like Lasso and Random Forests, when combined with carefully engineered linguistic and clinical features, surprisingly outperformed complex language models.

What Problem Does This Paper Address?

Standardized medical exams, like the USMLE, are high-stakes gatekeepers for healthcare professionals. To ensure these exams accurately assess competency, question difficulty and the time allotted for each question must be carefully balanced. However, manually designing and calibrating these questions is a complex and time-consuming process. Prior research has shown that predicting both question difficulty and the average time test-takers spend on them is a challenging task. To address this, the National Board of Medical Examiners (NBME) hosted a shared task, providing researchers with retired USMLE questions and their associated difficulty and response time data. The goal was to develop systems that could predict these metrics for new, unseen questions.

Methods

The ITEC team employed a multi-faceted approach, combining statistical models, machine learning, and language models. Their methodology involved several key steps:

  • Feature Extraction: They extracted a rich set of features from the questions, including:
  • Raw Features: Basic information like answer key, item type, exam step, and number of answer options.
  • Linguistic Features: Using tools like Linguistic Inquiry and Word Count (LIWC-22) to analyze word usage, sentence structure, and psychological dimensions, and TAALES 2.2 for lexical sophistication.
  • Clinical Embeddings: Leveraging pre-trained biomedical language models like Bio_ClinicalBERT to generate vector representations of the question text, capturing clinical context.
  • Model Development: They experimented with various models:
  • Statistical Models: Including Lasso regression, using stepwise regression and information criteria (AIC/BIC) for feature selection.
  • Machine Learning Models: Random Forest, Support Vector Machines (SVM), and K-Nearest Neighbors (KNN), often preceded by Principal Component Analysis (PCA) for dimensionality reduction.
  • Language Models: Fine-tuning domain-specific pre-trained models like BERT-ClinicalQA and Clinical-Longformer for a multi-target regression task (predicting both difficulty and response time simultaneously).
  • Ensemble Model: Combining predictions from multiple individual models to potentially improve robustness.

Datasets

The study utilized a dataset of retired USMLE multiple-choice questions provided by the NBME.

  • Training Phase: 466 questions were provided with metadata, item difficulty, and average response time.
  • Evaluation Phase: An additional 201 questions were provided without the target difficulty and response time labels.

The paper does not introduce a new benchmark dataset but rather uses the provided NBME data for the shared task.

Key Insights and Ideas

  • Simplicity Wins (Sometimes): Surprisingly, simpler models like Lasso and Random Forests, when applied to carefully engineered features (especially principal components of embeddings), often outperformed complex, fine-tuned language models. This aligns with Occam's Razor principle, suggesting that for this task, a more parsimonious approach was more effective.
  • Feature Engineering is Crucial: The quality and selection of features were paramount. Linguistic features (LIWC, TAALES) and clinical embeddings from BERT proved highly valuable, especially when combined with dimensionality reduction techniques.
  • Response Time is "Easier" to Predict: The study found that predicting response time was generally more straightforward than predicting difficulty. Features like item length, exam step, and analytical thinking were strong indicators for response time.
  • Difficulty Prediction is Stubborn: Predicting item difficulty proved more challenging, with most models performing close to a baseline. This suggests that capturing the nuanced cognitive processes involved in assessing difficulty requires more sophisticated modeling or features.
  • Ensemble Benefits: While not always the top performer, the ensemble model showed competitive results, highlighting the value of combining diverse predictive strengths.

Key Results

  • Difficulty Prediction:
    • The team's Random Forest model ranked 4th out of 43 submissions on the leaderboard for difficulty prediction.
  • Their Ensemble model ranked 12th out of 43 submissions.
  • Post-hoc analysis revealed that if the Lasso model had been submitted for difficulty prediction, it would have secured the 2nd position.
  • Response Time Prediction:
    • The team's Lasso model achieved the 2nd position out of 34 submissions for response time prediction.
  • The Lasso model's RMSE was 24.116, significantly better than the baseline (31.68).
  • Feature Importance: For response time, word count from LIWC, exam step, item type, and analytical thinking were identified as key predictors. For difficulty, word count, BERT answer embeddings, and analytical thinking were also important.

Limitations and Future Work

  • Aggregated Data: The study was limited by the use of aggregated response data per item, rather than individual-level student responses. Access to individual data would allow for more advanced psychometric modeling and exploration of item-student interactions.
  • Hyperparameter Tuning: The authors acknowledge that a more exhaustive grid search for hyperparameter tuning, especially for fine-tuned language models, could potentially yield better results.
  • Feature Set: The study focused on a predetermined set of features (LIWC, TAALES, BERT). Future work could explore additional feature extraction methods.
  • Simpler Models Advantage: Further investigation into why simpler models excel in certain scenarios is warranted.
  • Deeper Clinical Text Representation: For difficulty prediction, more intricate modeling of deep clinical text representations might be necessary.

Summary updated

BibTeX
@inproceedings{tack2024itec,
  author = {Anaïs Tack and Siem Buseyne and Changsheng Chen and Robbe D’hondt and Michiel De Vrindt and Alireza Gharahighehi and Sameh Metwaly and Felipe Kenji Nakano and Ann-Sophie Noreillie},
  title = {{ITEC at BEA 2024 Shared Task: Predicting Difficulty and Response Time of Medical Exam Questions with Statistical, Machine Learning, and Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure