TL;DR
This paper investigates how well GPT-4 can automatically score short L2 English essays on the CEFR scale, comparing its performance to human raters and existing Automated Writing Evaluation (AWE) systems. The study finds that GPT-4, with just a few calibration examples, can achieve performance comparable to established AWE models, though its agreement with human raters varies across different first languages (L1s).
What Problem Does This Paper Address?
Evaluating L2 writing proficiency is crucial for high-stakes language assessments. While automated essay scoring (AWE) systems have been around for decades, human scoring is still considered the gold standard due to its perceived accuracy. However, human scoring is expensive and prone to issues like rater fatigue and bias. The advent of powerful Large Language Models (LLMs) like GPT-4 presents a new opportunity to improve automated scoring. This paper aims to assess GPT-4's capabilities in this domain, specifically its ability to assign CEFR levels to short L2 essays, and to understand its fairness and agreement with human judgments.
Methods
The researchers employed a multi-pronged approach to evaluate GPT-4's essay scoring abilities.
- Calibration and Minimal Rubric: Initially, GPT-3.5 and GPT-4 were tasked with scoring essays using only a minimal rubric and varying numbers of calibration examples. Calibration examples are pre-scored essays that help the model understand the desired rating criteria.
- Prompt Engineering: To enhance performance, two prompt engineering strategies were tested:
- Detailed Rubric: Providing GPT-4 with a more comprehensive rubric detailing the criteria for each CEFR level.
- Require Rationale: Instructing GPT-4 to provide a step-by-step reasoning for its rating before assigning the score.
- Fairness Evaluation: The study also investigated GPT-4's fairness by analyzing its scoring performance across different genders and L1 backgrounds of the test-takers.
Datasets
The study utilized a human-rated dataset of short essay responses collected as part of the Duolingo English Test (DET). For this task, test-takers had 5 minutes to respond to a prompt. The dataset was carefully curated, controlling for L1 and gender, and focusing on 7 common L1 languages. To ensure representation across CEFR levels, a CEFR classifier was used to estimate levels, and then an equal number of responses were sampled for each L1, gender, and estimated CEFR level combination. The dataset comprised approximately 10,000 test sessions.
Key Insights and Ideas
- GPT-4's Learning Curve: GPT-4's performance significantly improves with even a small number of calibration examples. Without them, it underperforms even simple baselines.
- Near-State-of-the-Art Performance: With just one calibration example per rating category, GPT-4 achieved performance comparable to established AWE models, a remarkable feat for a general-purpose LLM.
- Prompt Engineering Nuances: While detailed rubrics and rationale requests are common prompt engineering techniques, their impact on GPT-4's performance was minimal if at least one calibration example per category was already provided. This suggests calibration is the primary driver of accuracy.
- Fairness Considerations: GPT-4 did not exhibit bias based on gender. However, its agreement with human raters varied significantly depending on the test-taker's L1, with lower agreement observed for certain languages.
Key Results
- Calibration is Key: GPT-4's Quadratic Weighted Kappa (QWK) with human raters jumped from below the length-only baseline to nearly matching the AWE baseline (0.81 vs. 0.84) with just one calibration example per category.
- GPT-3.5 Lagging: GPT-3.5 showed much slower improvement with calibration and only slightly outperformed the length-only baseline.
- CEFR Level Distribution: When calibrated, GPT-4 learned to use the full range of CEFR levels, but still struggled to differentiate between adjacent levels compared to humans, particularly for B2.
- L1 Agreement Variation: GPT-4's QWK with human raters ranged from 0.66 (Telugu) to 0.89 (Spanish). This variation was partly explained by differences in the distribution of human ratings for those L1s.
Limitations and Future Work
The authors acknowledge that the observed differences in agreement by L1 are not fully understood. It's unclear whether these discrepancies are due to the reliability of GPT-4's ratings or the inherent reliability of human ratings for certain L1s. The dataset did not contain enough double-rated essays to definitively investigate this.
Future work could explore:
- Advanced Prompt Engineering: Investigating other prompt engineering strategies.
- Fine-tuning: Leveraging the vast amount of data available for fine-tuning LLMs like GPT-3.5 to potentially achieve even better performance.
- Feedback Generation: A particularly exciting avenue is exploring GPT-4's potential for providing constructive feedback aligned with essay scoring, a task for which it appears well-suited.
- Deeper L1 Analysis: Further research is needed to understand the L1-specific agreement variations and their implications for fairness.
Summary updated