Automated Essay Scoring: Recent Successes and Future Directions

Shengjie Li, Vincent NgIJCAI 2024

Introduction

Automated Essay Scoring (AES) is the task of using natural language processing (NLP) to assign scores to essays that reflect their quality. While it may appear to be a straightforward regression or classification problem, AES is deceptively complex. This survey paper provides a comprehensive overview of the field, tracing its evolution from early rule-based heuristics to modern Transformer-based architectures and Large Language Models (LLMs).

The authors identify two primary hurdles currently facing the field:

  1. Holistic vs. Trait-Specific Scoring: Most systems provide a single "holistic" score. However, for educational utility, students require feedback on specific traits like Organization, Coherence, and Argument Persuasiveness. Scoring these content-based traits is significantly more difficult than identifying non-content issues like grammar or spelling.
  2. The Generalization Gap: While models perform well when trained and tested on the same essay prompt (within-prompt scoring), their performance often plummets when applied to new, unseen topics (cross-prompt scoring). This limitation hinders their application in dynamic classroom environments.

Methodological Evolution

The paper categorizes the evolution of AES into three distinct eras:

  • Heuristic Approaches (Early years – 2010s): Early systems like e-rater relied on hand-crafted rules to identify structural elements. While efficient, these systems struggled to grasp the actual meaning of the text.
  • Machine Learning Approaches (2010s): The field shifted toward training models (such as SVMs) on extensive sets of engineered features, including n-grams, syntactic patterns, and readability metrics.
  • Neural & Transformer Approaches (Current): Modern state-of-the-art systems involve fine-tuning pre-trained models like BERT or RoBERTa. These models often use a combination of regression loss for score accuracy and ranking loss to ensure relative essay quality is preserved. To address the generalization gap, researchers employ domain adaptation, contrastive learning, and prompt-independent features. The newest frontier involves using LLMs (like GPT-4) in zero-shot or few-shot settings by providing scoring rubrics directly in the system prompt.

Datasets

The field is heavily reliant on a few key corpora, most notably the ASAP dataset. These datasets provide the foundation for training and benchmarking various AES models.

Overview of prominent AES corpora, detailing writer demographics, essay counts, and scoring ranges.
  • ASAP (Automated Student Assessment Prize): The most widely used dataset, containing approximately 17,000 essays from US 7th to 10th graders.
  • ASAP++: An extension of ASAP that includes scores for multiple traits, enabling research into explainable feedback.
  • TOEFL11: Essays from non-native English speakers across 11 different native languages, primarily used for proficiency-based scoring.

Key Results and Insights

The standard metric for AES is Quadratic Weighted Kappa (QWK), which measures agreement between the system and human raters on a 0 to 1 scale.

Performance metrics (QWK) for holistic and trait-specific scoring across various model settings.

As shown in the performance data, there is a significant drop in accuracy when moving from within-prompt to cross-prompt settings. For instance, the NPCR model achieves a QWK of 0.82 within-prompt, while the best cross-prompt model (PANN) reaches only 0.70. Furthermore, trait-specific scoring remains a challenge; scores for traits like Conventions (0.56) and Voice (0.58) are generally lower than holistic scores.

Key insights from the research include:

  • The "Content" Bottleneck: Non-content traits (Grammar, Fluency) are relatively easy to automate. Content-based traits (Persuasiveness) require deep semantic understanding that current neural networks struggle to achieve without massive labeled data.
  • LLMs are not a "Silver Bullet": Preliminary experiments show that state-of-the-art LLMs in zero-shot settings often perform worse than smaller, supervised models specifically trained for AES, as LLMs may interpret rubrics differently than human raters.
  • Multi-Task Learning (MTL) for Explainability: Training models to predict both holistic and individual trait scores simultaneously is a promising path toward "Explainable AES," forcing models to reason about their grades and provide actionable feedback.

Future Directions

The authors highlight several areas for improvement:

  • Data Stagnation: There is an urgent need for newer, more diverse corpora beyond the decade-old ASAP dataset.
  • Extrinsic Evaluation: Research should move beyond matching human scores (intrinsic evaluation) to seeing if these systems actually help students improve their writing over time (extrinsic evaluation).
  • Rigorous Error Analysis: Instead of just reporting higher QWK scores, researchers should explain which specific types of essays or traits their models handle better.

Summary updated

BibTeX
@inproceedings{li2024automated,
  author = {Shengjie Li and Vincent Ng},
  title = {{Automated Essay Scoring: Recent Successes and Future Directions}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure