The British Council submission to the BEA 2024 shared task

Mariano Felice, Zeynep Duran KaraozNAACL 2024

TL;DR

This paper presents the British Council's submission to the BEA 2024 shared task on predicting item difficulty for medical multiple-choice questions (MCQs). They found that simple linear regression models, especially when ensembled, can achieve competitive results, even outperforming more complex transformer-based models in some cases, highlighting the importance of feature engineering.

What Problem Does This Paper Address?

Creating high-stakes exams like those for medical licensing is a complex and costly process. A crucial aspect is determining the difficulty of individual test items to ensure fairness and validity. Traditionally, this involves pre-testing items on a large sample of test-takers, which is time-consuming and resource-intensive. This paper addresses the challenge of automating item difficulty prediction using Natural Language Processing (NLP) techniques, aiming to reduce costs and improve efficiency in test development.

Methods

The authors explored three main approaches:

  • Feature-based Linear Regression: They extracted over a hundred linguistic features from the MCQs, covering lexical, semantic, syntactic, cognitive, and frequency aspects. They experimented with different feature selection methods (e.g., SelectKBest, Random Forest) and regression algorithms (Linear Regression, SGD Regressor, SVR, etc.).
  • Transformer Models: They fine-tuned pre-trained language models (BERT, RoBERTa, bioBERT, XLNet) by adding a linear layer to predict difficulty. They also experimented with incorporating the linguistic features alongside the transformer embeddings.
  • Ensemble Models: They combined the predictions of their best feature-based and transformer models using simple averaging, minimum/maximum selection, and a linear regression stacking model.

Datasets

The study utilized the shared task dataset, which consists of 667 retired Multiple Choice Questions (MCQs) from past administrations of the United States Medical Licensing Examination (USMLE). The dataset was split into training (466 items) and testing (201 items) sets. For model development and hyperparameter tuning, the training data was further split into an 80%-20% train-development set.

Key Insights and Ideas

  • Simplicity Can Be Powerful: Surprisingly, a well-tuned linear regression model using carefully selected linguistic features achieved excellent performance, even outperforming some transformer-based models. This suggests that for this specific task and dataset size, explicit feature engineering can be as, or even more, effective than end-to-end deep learning.
  • Ensembling Works: The ensemble model, combining the strengths of the feature-based and transformer models, achieved the best overall performance, ranking 5th in the shared task. This underscores the benefit of combining diverse predictive approaches.
  • Middle-Ground Accuracy: The ensemble model was most accurate for items in the middle of the difficulty range, struggling more with very easy or very difficult items. This indicates a potential bias in the model's learning or the data distribution.
  • Feature Importance: The explicit definition of linguistically-motivated features was crucial for achieving competitive results, especially when compared to models that relied solely on raw text embeddings.

Key Results

The British Council's submission included three models: a feature-based linear regressor (FEAT), a RoBERTa-based model (ROBERTA), and a linear regression ensemble (ENSEMBLE).

  • Official Rankings: The ENSEMBLE model ranked 5th with an RMSE of 0.305, FEAT ranked 7th with an RMSE of 0.305, and ROBERTA ranked 8th with an RMSE of 0.306.
  • Baseline Beating: All three models outperformed the baseline DummyRegressor (RMSE = 0.311), with only 35% of all submitted systems achieving this.
  • Feature Selection Impact: The best feature-based model used 10 features, including readability measures, cognitively-motivated features, and word frequency counts.
  • Transformer Performance: RoBERTa showed the best performance among the transformer models, but even with extensive hyperparameter tuning, it did not surpass the initial feature-based linear regression model.

Limitations and Future Work

  • Data Size: The authors acknowledge that the small size of the training data may have hindered the performance of all participating systems.
  • Difficulty Range: The ensemble model showed less accuracy for very easy or very difficult items.
  • Future Directions: The authors suggest exploring custom loss functions for transformer models, incorporating new features, and generating synthetic data to overcome the limitations of the small dataset. They also note that while pictures and the number of distractors were considered, they didn't significantly impact prediction accuracy in their models.

Summary updated

BibTeX
@inproceedings{felice2024british,
  author = {Mariano Felice and Zeynep Duran Karaoz},
  title = {{The British Council submission to the BEA 2024 shared task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure