Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction

Rishikesh Fulari, Jonathan RusertNAACL 2024

TL;DR

This paper explores using machine learning to predict the difficulty and time it takes to answer medical exam questions. By leveraging linguistic features and BERT embeddings, they achieved competitive results, paving the way for more efficient and fair test construction by reducing reliance on traditional, labor-intensive pretesting.

What Problem Does This Paper Address?

Creating fair and effective standardized tests is a complex challenge. A key aspect is understanding how difficult questions are and how long students will take to answer them. Traditionally, this is done through pretesting, where new questions are piloted on actual exams. However, this is costly, time-consuming, and can compromise test security if questions are leaked. This paper tackles the "cold-start parameter estimation problem" – how to predict these item characteristics (difficulty and response time) directly from the question text itself, thereby streamlining the process and enhancing fairness.

Methods

The researchers experimented with two main feature engineering approaches:

  • Embeddings: They used sentence Transformer embeddings, specifically a BERT model fine-tuned on medical text (PubMedBERT) and then further adapted for information retrieval (MS-MARCO dataset). This converts the entire question text into a 768-dimensional vector.
  • Linguistic Features: This approach extracts specific linguistic characteristics from the text. These include word count, number of unique words, counts of additives, numbers, and letters. They also incorporated metadata like ItemType (whether the question had an image) and EXAM (the specific USMLE exam level). Each question was encoded into a 9-dimensional vector using these features.

They then applied various machine learning models to these features for both predicting difficulty and response time. These models included classical approaches like Linear Regression, Decision Trees, and K-Nearest Neighbors, as well as ensemble methods like Random Forest, XGBoost, and Gradient Boosting Decision Trees (GBDT).

Datasets

The study utilized a dataset of 667 previously used and retired Multiple Choice Questions (MCQs) from the United States Medical Licensing Examination (USMLE) Steps 1, 2 CK, and 3. The training set comprised 466 questions, and the test set contained 201. The item characteristics (difficulty and response time) were derived from first-time examinees from accredited US and Canadian medical schools.

Key Insights and Ideas

  • Linguistic Features Shine for Response Time: The paper found that linguistic features were particularly effective for predicting response time. This aligns with the intuition that more complex or longer questions, characterized by higher word counts and specific linguistic structures, naturally take longer to process.
  • Embeddings vs. Linguistic Features Trade-off: While BERT embeddings offer a rich representation, the reduction in dimensionality achieved by linguistic features (from 768 to 9 dimensions) significantly improved the performance of simpler models like Linear Regression, suggesting that for this specific task, the "curse of dimensionality" was a real factor.
  • Model and Feature Synergy: The best-performing model wasn't a single universal winner. For difficulty prediction, a fine-tuned BERT model initially showed promise, but ultimately, Linear Regression with linguistic features achieved the lowest RMSE. For response time, the Random Forest Regressor with linguistic features performed best.

Key Results

  • Difficulty Prediction: The best model for predicting item difficulty achieved a Root Mean Squared Error (RMSE) of 0.302 using Linear Regression with linguistic features.
  • Response Time Prediction: The best model for predicting response time achieved an RMSE of 26.181 using Linear Regression with linguistic features. The Random Forest Regressor with linguistic features was very close, with an RMSE of 26.234.
  • Ranks: The authors' best model (Random Forest Regressor on Linguistic Features) secured 14th rank out of 43 for Item Difficulty Prediction and 9th rank out of 34 for Response Time Prediction in the BEA Shared Task.

Limitations and Future Work

  • Subset vs. Full Training Data: The initial promising results for BERT were based on a subset of the data for validation and testing. When trained on the entire dataset, other models, particularly those using linguistic features, surpassed BERT for the response time task.
  • Generalizability: While the USMLE dataset is a high-stakes medical exam, the findings might not directly translate to other domains or question types without further validation.
  • Future Directions: The authors suggest exploring additional techniques and validating their findings across diverse educational contexts. They also highlight the potential of their approach to streamline assessment and improve fairness by reducing reliance on labor-intensive pretesting.

Summary updated

BibTeX
@inproceedings{fulari2024utilizing,
  author = {Rishikesh Fulari and Jonathan Rusert},
  title = {{Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure