UNED team at BEA 2024 Shared Task: Testing different Input Formats for predicting Item Difficulty and Response Time in Medical Exams

Alvaro Rodrigo, Sergio Moreno-Álvarez, Anselmo PeñasNAACL 2024

TL;DR

This paper explores using BERT-based models to predict the difficulty and response time of medical exam questions. Surprisingly, the best difficulty predictions came from using only the clinical text, while response time predictions benefited from including the correct answer, highlighting distinct challenges for each prediction task.

What Problem Does This Paper Address?

Ensuring the validity and fairness of assessments hinges on accurately calibrating exam difficulty to the intended evaluation level. This allows educators to gauge student understanding and proficiency effectively, promoting an equitable learning environment. The BEA 2024 Shared Task specifically addresses the challenge of automatically predicting the difficulty and response time for standardized medical exams, aiming to build a common framework for this research. This is particularly relevant as educators increasingly seek to leverage language technologies for such applications.

Methods

The UNED team employed BERT-base models, fine-tuned for regression, as their core technology. They focused on simpler, computationally less demanding versions of these transformer models. The key innovation in their approach was the systematic exploration of different input formats to understand the impact of various text elements on prediction accuracy. They experimented with three main input strategies: concatenating all text elements (clinical case, question, and all answer options), using only the clinical text and the correct answer, and using solely the clinical text and question. Pre-processing involved minimal steps, primarily using the BERT tokenizer and scaling target variables to a [0, 1] range using MinMaxScaler.

Datasets

The study utilized data from the United States Medical Licensing Examination, provided for the BEA 2024 Shared Task. This dataset consists of multiple-choice medical exam questions with answer candidates ranging from 4 to 10. The data was split into training and development sets, with 10% of the training collection used for development experiments.

Key Insights and Ideas

  • Input Format Matters, Differently for Each Task: The most striking finding is how the optimal input format diverged significantly between predicting item difficulty and response time. This suggests these are fundamentally different prediction problems, requiring distinct feature engineering.
  • Clinical Text Alone is Key for Difficulty: For predicting item difficulty, the simplest input format – just the clinical case and question text – yielded the best results. This implies that extraneous information from answer choices might actually hinder accurate difficulty prediction in this context.
  • Correct Answer Aids Response Time Prediction: Conversely, including the correct answer alongside the clinical text significantly improved response time predictions, indicating that the content of the correct answer plays a crucial role in estimating how long a candidate might take.
  • BERT-based Models are Promising: Despite using relatively standard BERT models and simple input strategies, the team achieved competitive results, particularly excelling in response time prediction.

Key Results

In Track 1 (Item Difficulty Prediction), the UNED team's best submission (Run 3, using only clinical text) achieved a Root Mean Squared Error (RMSE) of 0.308, ranking 13th overall. While not winning, this run was close to the top systems and outperformed the proposed baseline. Run 1, which included all answer candidates, ranked 35th.

In Track 2 (Response Time Prediction), the UNED team demonstrated strong performance. Their best submission (Run 2, using clinical text and the correct answer) achieved the top rank with an RMSE of 23.927. Run 1 (all text) ranked 3rd, and Run 3 (only text) ranked 4th, showcasing the effectiveness of their approaches across different input formats for this task.

Limitations and Future Work

The authors acknowledge that their performance in difficulty prediction was only moderate, ranking in the middle of participants. They suggest that a more profound study is needed on how to correctly incorporate answer candidates for difficulty prediction. Future work aims to explore new configurations, include more systems in the study, and investigate more complex approaches beyond BERT-base models. They also plan to study the impact of using only the clinical text and question, including the correct answer, and including all candidates, to further refine their understanding of these prediction tasks.

Summary updated

BibTeX
@inproceedings{rodrigo2024uned,
  author = {Alvaro Rodrigo and Sergio Moreno-Álvarez and Anselmo Peñas},
  title = {{UNED team at BEA 2024 Shared Task: Testing different Input Formats for predicting Item Difficulty and Response Time in Medical Exams}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure