Introduction
Automated Essay Scoring (AES) is the task of using natural language processing (NLP) to assign scores to essays that reflect their quality. While it may appear to be a straightforward regression or classification problem, AES is deceptively complex. This survey paper provides a comprehensive overview of the field, tracing its evolution from early rule-based heuristics to modern Transformer-based architectures and Large Language Models (LLMs).
The authors identify two primary hurdles currently facing the field:
- Holistic vs. Trait-Specific Scoring: Most systems provide a single "holistic" score. However, for educational utility, students require feedback on specific traits like Organization, Coherence, and Argument Persuasiveness. Scoring these content-based traits is significantly more difficult than identifying non-content issues like grammar or spelling.
- The Generalization Gap: While models perform well when trained and tested on the same essay prompt (within-prompt scoring), their performance often plummets when applied to new, unseen topics (cross-prompt scoring). This limitation hinders their application in dynamic classroom environments.
Methodological Evolution
The paper categorizes the evolution of AES into three distinct eras:
- Heuristic Approaches (Early years – 2010s): Early systems like e-rater relied on hand-crafted rules to identify structural elements. While efficient, these systems struggled to grasp the actual meaning of the text.
- Machine Learning Approaches (2010s): The field shifted toward training models (such as SVMs) on extensive sets of engineered features, including n-grams, syntactic patterns, and readability metrics.
- Neural & Transformer Approaches (Current): Modern state-of-the-art systems involve fine-tuning pre-trained models like BERT or RoBERTa. These models often use a combination of regression loss for score accuracy and ranking loss to ensure relative essay quality is preserved. To address the generalization gap, researchers employ domain adaptation, contrastive learning, and prompt-independent features. The newest frontier involves using LLMs (like GPT-4) in zero-shot or few-shot settings by providing scoring rubrics directly in the system prompt.
Datasets
The field is heavily reliant on a few key corpora, most notably the ASAP dataset. These datasets provide the foundation for training and benchmarking various AES models.
- ASAP (Automated Student Assessment Prize): The most widely used dataset, containing approximately 17,000 essays from US 7th to 10th graders.
- ASAP++: An extension of ASAP that includes scores for multiple traits, enabling research into explainable feedback.
- TOEFL11: Essays from non-native English speakers across 11 different native languages, primarily used for proficiency-based scoring.
Key Results and Insights
The standard metric for AES is Quadratic Weighted Kappa (QWK), which measures agreement between the system and human raters on a 0 to 1 scale.
As shown in the performance data, there is a significant drop in accuracy when moving from within-prompt to cross-prompt settings. For instance, the NPCR model achieves a QWK of 0.82 within-prompt, while the best cross-prompt model (PANN) reaches only 0.70. Furthermore, trait-specific scoring remains a challenge; scores for traits like Conventions (0.56) and Voice (0.58) are generally lower than holistic scores.
Key insights from the research include:
- The "Content" Bottleneck: Non-content traits (Grammar, Fluency) are relatively easy to automate. Content-based traits (Persuasiveness) require deep semantic understanding that current neural networks struggle to achieve without massive labeled data.
- LLMs are not a "Silver Bullet": Preliminary experiments show that state-of-the-art LLMs in zero-shot settings often perform worse than smaller, supervised models specifically trained for AES, as LLMs may interpret rubrics differently than human raters.
- Multi-Task Learning (MTL) for Explainability: Training models to predict both holistic and individual trait scores simultaneously is a promising path toward "Explainable AES," forcing models to reason about their grades and provide actionable feedback.
Future Directions
The authors highlight several areas for improvement:
- Data Stagnation: There is an urgent need for newer, more diverse corpora beyond the decade-old ASAP dataset.
- Extrinsic Evaluation: Research should move beyond matching human scores (intrinsic evaluation) to seeing if these systems actually help students improve their writing over time (extrinsic evaluation).
- Rigorous Error Analysis: Instead of just reporting higher QWK scores, researchers should explain which specific types of essays or traits their models handle better.
Summary updated