TL;DR
This paper introduces LCES, a novel zero-shot automated essay scoring (AES) method that leverages Large Language Models (LLMs) to perform pairwise comparisons between essays, rather than directly predicting absolute scores. This approach significantly improves accuracy and robustness compared to existing zero-shot methods, offering a more reliable and cost-effective solution for essay evaluation.
What Problem Does This Paper Address?
Automated Essay Scoring (AES) aims to automate the grading of essays, reducing the cost and effort associated with manual evaluation. Traditional AES methods often require large datasets of scored essays for each specific prompt, which is expensive and time-consuming to collect. While recent advancements in LLMs have enabled "zero-shot" AES (scoring without prior training data), these methods typically rely on LLMs to directly assign absolute scores. However, this direct scoring approach is prone to LLM biases, inconsistent scoring, and a general divergence from human evaluations. The core challenge this paper addresses is the development of a zero-shot AES method that overcomes these limitations, achieving high accuracy and reliability without requiring labeled training data for each prompt.
Methods
The proposed method, LCES (LLM-based Comparative Essay Scoring), reframes AES as a pairwise comparison task. Instead of asking an LLM to assign a score, LCES instructs the LLM to determine which of two essays is better, given a prompt and rubric. This pairwise comparison data is then used to train a RankNet model. RankNet is a neural network architecture designed to learn latent scores from pairwise preferences.
The process involves:
- Pairwise Comparison Generation: LLMs are prompted to compare pairs of essays. To mitigate position bias (where the order of essays affects the LLM's judgment), each pair is compared twice, and the results are debiased. Chain-of-thought prompting is used to encourage the LLM to provide reasoning for its decision.
- Latent Score Estimation: A RankNet model, with shared weights for processing both essays in a pair, learns to predict the probability of one essay being preferred over another. This model is trained using a binary cross-entropy loss function to minimize the discrepancy between predicted and actual pairwise preferences.
- Output Conversion: The learned latent scores from RankNet are then converted into standard AES outputs, such as rankings or scores within a defined rubric range, using linear transformation.
Datasets
The authors evaluated LCES on two widely used benchmark datasets for AES:
- ASAP (Automated Student Assessment Prize): This dataset contains approximately 13,000 essays across eight different prompts, each with human-assigned scores.
- TOEFL11: This dataset comprises about 12,100 essays written by non-native English speakers, also with human-assigned scores.
No new benchmarks were introduced; the focus was on evaluating LCES against existing methods on established datasets.
Key Insights and Ideas
- Pairwise Comparison as a Superior Paradigm: The core insight is that framing AES as a pairwise comparison task, rather than direct score prediction, significantly improves LLM performance and robustness. LLMs seem to be more adept at relative judgments than absolute scoring.
- RankNet for Scalable Latent Score Learning: The use of RankNet is crucial for efficiently converting a large number of pairwise comparisons into meaningful latent scores, overcoming the quadratic complexity of comparing all possible essay pairs.
- Robustness Across LLMs: LCES demonstrates consistent performance across a variety of LLM backbones, indicating its general applicability and reducing reliance on specific, high-performing models.
- Position Bias Mitigation: The simple yet effective debiasing strategy for pairwise comparisons significantly improves reliability, especially for LLMs that are more sensitive to essay order.
Key Results
- Outperforms Zero-Shot Baselines: LCES consistently outperformed conventional zero-shot AES methods (Vanilla and MTS) in terms of Quadratic Weighted Kappa (QWK) scores across both ASAP and TOEFL11 datasets. For instance, on ASAP, LCES with Llama-3.1-8B achieved an average QWK of 0.670, compared to 0.633 for MTS and 0.426 for Vanilla.
- Competitive with Supervised Methods: LCES with Llama-3.1-8B achieved QWK scores comparable to supervised learning methods on the ASAP dataset, narrowing the performance gap significantly.
- Low Variance: LCES exhibited significantly lower performance variance across different LLM backbones compared to MTS and Vanilla, suggesting greater stability and predictability.
- Effective Position Bias Correction: The position bias correction method showed notable improvements for LLMs with higher inconsistency rates, demonstrating its value in enhancing reliability.
- Data Efficiency: RankNet-based LCES achieved high QWK scores even with a limited number of pairwise comparisons, highlighting its data efficiency.
Limitations and Future Work
- LLM-Generated Noise: LCES relies on LLM-generated pairwise labels, which may contain noise or inconsistencies, directly impacting the quality of the learned scoring model.
- Determining Optimal Comparison Count: The paper acknowledges that it's unclear how to determine the optimal number of pairwise comparisons (M) needed for reliable scoring, limiting systematic control over scoring quality.
- Absolute Scale Transformation: Mapping latent scores to an absolute scale via linear transformation assumes the sampled comparisons cover the full score range. If certain score ranges are underrepresented, this can lead to inaccurate absolute scores.
- Zero-Shot Performance Assessment: The zero-shot nature of LCES means its performance cannot be quantitatively assessed without labeled data, necessitating caution for high-stakes applications.
- Future Work: The authors suggest further auditing for latent biases in LLMs beyond position bias, and emphasize the need for thorough validation of LCES on specific prompts and rubrics before real-world deployment. They also highlight LCES's potential for inductive settings, where new essays can be scored without retraining.
Summary updated