Predicting Item Difficulty and Item Response Time with Scalar-mixed Transformer Encoder Models and Rational Network Regression Heads

Sebastian Gombert, Lukas Menzel, Daniele Di Mitri, Hendrik DrachslerNAACL 2024

TL;DR

This paper tackles the challenge of automatically predicting item difficulty and response time for medical licensing exam questions. The authors developed a novel multi-task learning architecture using transformer encoders, scalar mixing, and rational activation functions, achieving first place in predicting item difficulty and fifth in response time in a shared task.

What Problem Does This Paper Address?

Assessing the difficulty and response time of educational assessment items is crucial for understanding learner performance and making informed educational decisions. Traditionally, this has relied on labor-intensive pre-evaluation with cohorts of test-takers. This paper addresses the need for automated methods to predict these item characteristics, specifically for multiple-choice questions from the United States Medical Licensing Examination (USMLE). The authors highlight a research gap in applying modern transformer-encoder language models, like BERT, to this problem, aiming to improve upon existing feature-based approaches.

Methods

The core of the proposed solution is a multi-task learning architecture built upon pre-trained transformer encoder models. Key innovations include:

  • Scalar Mixing: Instead of using only the final layer's output from the transformer, this technique calculates a weighted average of hidden layer representations from all layers. The weights are learned during training, allowing the model to dynamically determine the importance of different linguistic phenomena captured at various depths of the transformer.
  • Rational Activation Functions: The regression heads employ a two-layer structure with "Rational Activations." These are learnable activation functions based on Padé approximants, which can flexibly approximate various functions and have shown superior performance to standard activations in other tasks.
  • Multi-Task Learning: The model is trained to predict both item difficulty and response time simultaneously. This leverages the correlation between the two variables, allowing the shared transformer encoder to learn representations beneficial for both prediction tasks. Separate regression heads are used for each task, allowing for distinct learning dynamics.

Datasets

The primary dataset used is from the BEA 2024 Shared Task on Automated Prediction of Difficulty and Response Time. This dataset consists of multiple-choice items from the United States Medical Licensing Examination (USMLE). It was divided into a training set of 466 items and a test set of 201 items.

Key Insights and Ideas

  • Transformer Power for Educational Assessment: The study demonstrates that advanced transformer models, when adapted with techniques like scalar mixing and rational activations, can achieve state-of-the-art performance in predicting item difficulty and response time, outperforming traditional methods.
  • Scalar Mixing's Value: The ability to leverage representations from all transformer layers, rather than just the last, proved crucial. This suggests that different levels of linguistic abstraction within the transformer are important for predicting item characteristics.
  • Rational Activations' Flexibility: The use of learnable rational activation functions in the regression heads allowed for more flexible modeling of the complex relationships between item features and the target variables.
  • Item Difficulty is Stubbornly Hard: Despite strong performance in other areas, predicting item difficulty proved significantly more challenging, with even the best models barely outperforming a simple dummy regressor. This suggests inherent complexities or limitations in the available data.
  • Response Time is More Predictable: Predicting item response time yielded much better results, with transformer models significantly outperforming baselines. This indicates that the factors influencing how long a test-taker spends on an item are more readily captured by language models.

Key Results

The authors' best model achieved:

  • First place out of 43 submissions for predicting Item Difficulty.
  • Fifth place out of 34 submissions for predicting Item Response Time.

While the exact RMSE scores are detailed in the tables, the key takeaway is the strong performance of their transformer-based approach, especially in contrast to baseline models. For Item Difficulty, even the best models had minimal predictive power compared to a dummy regressor, but their correlation coefficients (r and rs) were still superior to baselines. For Item Response Time, models significantly outperformed baselines, with DeBERTa-v3-large showing the best performance in pre-evaluation for this metric.

Limitations and Future Work

  • Domain Specificity: The dataset is exclusively from the biomedical domain, which has specialized language. The authors acknowledge that results may not generalize to other domains, and cross-domain evaluations are needed.
  • Difficulty Prediction Remains Challenging: The difficulty in predicting item difficulty, even with advanced models, suggests that current approaches and data might be insufficient.
  • Future Work: The authors suggest integrating their approach with existing feature-based models by fine-tuning transformers and using their intermediate layer outputs as embeddings. They also propose evaluating models on datasets from diverse domains to assess generalizability and further explore the potential of task-specific transformer embeddings.

Summary updated

BibTeX
@inproceedings{gombert2024predicting,
  author = {Sebastian Gombert and Lukas Menzel and Daniele Di Mitri and Hendrik Drachsler},
  title = {{Predicting Item Difficulty and Item Response Time with Scalar-mixed Transformer Encoder Models and Rational Network Regression Heads}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure