TL;DR
This paper tackles the inefficiency of Comparative Judgment (CJ) assessments by proposing a "warm start" method. By using deep learning models to predict initial essay quality scores, they significantly reduce the number of judgments needed for reliable assessments, making CJ more practical for large-scale educational evaluations.
What Problem Does This Paper Address?
Comparative Judgment (CJ) is a powerful method for assessing student essays, offering flexibility and reliability by having multiple assessors compare pairs of essays. However, a major drawback is its inefficiency: assessors need to make a large number of comparisons (around 10-14 per essay) to achieve reliable scores. This is largely due to the "cold-start problem" – at the beginning of an assessment, there's no information about essay quality, making initial judgments less informative. Existing solutions to speed up CJ often have their own limitations. This paper aims to solve this cold-start problem by predicting initial essay quality scores, thereby providing a "warm start" and improving assessment efficiency.
Methods
The core of this research lies in two main components:
- Predicting Initial Essay Quality Scores: The authors employed deep learning, specifically fine-tuning pre-trained Dutch transformer models (BERTje, RobBERT, and RobBERTje). They experimented with two configurations:
- Using only the essay text as input.
-
Using both the essay text and the assignment prompt as input. The models were trained on data from existing CJ assessments to predict quality scores that would resemble the final scores obtained from a full CJ assessment.
-
Simulating CJ Assessments with a Warm Start: Once the best model for predicting initial scores was identified, the researchers simulated CJ assessments. They compared two scenarios:
- Cold-start BTL model: A standard Bayesian Bradley-Terry-Luce (BTL) model with a non-informative prior distribution for quality scores.
- Warm-start BTL model: The same Bayesian BTL model, but with prior distributions for quality scores informed by the predicted initial scores from the deep learning models. They then simulated the assessment process by iteratively adding judgments and measuring the reliability of the estimated quality scores using the Scale Separation Reliability (SSR) metric.
Datasets
The study utilized a dataset collected by Lesterhuis et al. (2022), comprising argumentative essays written in Dutch by 16-year-old students. These essays were from three different assignments: "Having children," "Organ donation," and "Stress experienced by students." The essays were already assessed using a comparative judgment method, providing the "final quality scores" used for training and evaluation. No new benchmark datasets were introduced.
Key Insights and Ideas
- Deep Learning for "Warm Starts" is Viable: The study demonstrates that deep learning models, particularly transformer architectures, can effectively predict initial essay quality scores that are highly correlated with final CJ scores. This validates the core idea of using AI to jumpstart the assessment process.
- Prompt Information Matters: Including the assignment prompt alongside the essay text significantly improved the reliability of predicted quality scores. This is crucial because essays in the training set might be from different assignments than those in the test set, making prompt context essential for accurate prediction.
- Significant Efficiency Gains: The warm-start approach dramatically reduces the number of judgments required to reach a target reliability level (0.70). This means assessors can achieve reliable results much faster, making CJ more practical.
- SSR as a Practical Reliability Metric: The study highlights the utility of the SSR metric for approximating reliability during an assessment, as true reliability is not known until the end. The warm-start approach shows faster SSR convergence.
- Early-Stage Ranking Accuracy: The warm-start method leads to more spread-out quality scores and better ranking accuracy from the very first judgments, providing more informative initial rankings compared to the cold-start approach.
Key Results
- The RobBERT model, when incorporating both essay text and prompt information, achieved the highest reliability in predicting quality scores.
- The warm-start approach reduced the number of judgments needed to reach a reliability of 0.70 by 35% to 41% for some assignments.
- For Assignment 2, a reliability of 0.70 was achieved with less than nine judgments per essay using the warm-start, compared to at least ten with a cold start (a 15% efficiency gain).
- The SSR metric showed a faster approximation of reliability when employing a warm start, reaching the 0.70 level for all assignments.
- Forest plots illustrating quality scores at different stages showed that the warm-start approach yielded more spread-out scores and better ranking accuracy from the outset.
Limitations and Future Work
- Model Training Data: The study relied on combining quality scores from different CJ assessments for training. This was feasible because the essays measured argumentative writing quality. However, if essays were from different genres (e.g., informative writing), combining scores would require calibration, and predicting scores for new genres might be more difficult.
- Simulation of Judgment Order: The simulations involved repeatedly shuffling judgment orders. While this had a negligible impact on reliability in their study, it might not perfectly reflect real-world CJ assessment processes where essays are typically compared more evenly.
- End-of-Assessment Reliability: The difference in reliability between warm and cold starts diminished after many judgments. Future work could explore methods for selecting essay pairs that maintain high reliability towards the end of an assessment, avoiding the pitfalls of adaptive selection rules.
- English vs. Dutch Models: The study used Dutch models for Dutch essays. The authors note that applying similar models to English essays might require evaluation, and unsupervised learning approaches could be considered if scored Dutch essays are unavailable for training.
- Alternative Scoring Methods: For scenarios where no prior scores are available for training deep learning models, alternative methods like unsupervised learning for AES should be explored.
Summary updated