Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards

Heejin Do, Sangwon Ryu, Gary Geunbae LeeEMNLP 2024 · DOI

Introduction

Automated Essay Scoring (AES) has traditionally focused on providing a single "holistic" grade. However, for educational purposes, students require nuanced feedback across multiple dimensions, such as organization, fluency, and grammar. This paper, presented at EMNLP 2024, introduces SaMRL (Scoring-aware Multi-reward Reinforcement Learning), a framework designed to provide high-quality multi-trait feedback by leveraging autoregressive models and reinforcement learning.

The Challenge: Metric Alignment and Model Scalability

The development of effective multi-trait AES systems faces two primary hurdles:

  1. The Metric Gap: The standard evaluation metric for AES is the Quadratic Weighted Kappa (QWK). Because QWK is non-differentiable, most neural models are trained using Mean Squared Error (MSE). This misalignment often leads to models that perform well on loss functions but fail to capture the grading nuances prioritized by human evaluators.
  2. Model Limitations: Traditional classification or regression models struggle with wide score ranges (e.g., 0–60) and often require separate output "heads" for each trait, which is computationally inefficient and ignores the dependencies between different grading criteria.

The SaMRL Framework

To address these issues, the authors propose treating essay scoring as a sequence generation task. Using a T5-based architecture, the model "writes" scores as a sequence of tokens (e.g., [Trait1 4, Trait2 3...]). This autoregressive approach allows a single model to capture all traits while modeling the correlations between them.

The SaMRL framework utilizes an autoregressive policy model and an anchor model with scoring-aware reward functions.

The core innovation is the Scoring-aware Multi-reward RL mechanism. Since QWK cannot be used as a direct loss function, it is employed as a reward within a Policy Gradient (PPO) framework. The system incorporates:

  • Bi-directional QWK Rewards: Rewards are calculated both across batches (comparing different essays) and across traits (ensuring internal consistency within a single essay).
  • Multi-objective Optimization: The reward function combines QWK with an MSE penalty for numerical precision and a KL-divergence term (using a frozen "anchor" model) to maintain output formatting and prevent model drift.

Experimental Results

The researchers evaluated SaMRL on the ASAP, ASAP++, and Feedback Prize benchmarks. SaMRL consistently achieved state-of-the-art performance, particularly on complex argumentative and narrative essays.

Performance comparison (QWK) of SaMRL against regression and classification baselines across eight prompts.

Key findings include:

  • Superior Accuracy: On the ASAP++ dataset, SaMRL-large achieved an average QWK of 0.705, outperforming previous autoregressive baselines like ArTS.
  • Robustness: The model showed significant improvements on prompts with wide score ranges (such as P7 and P8), where traditional classification models typically degrade due to the high number of possible classes.
  • Trait Correlation: By generating scores sequentially, the model effectively learned how different traits (like sentence fluency and conventions) relate to one another, capturing dependencies that independent scoring heads miss.

Conclusion and Future Work

SaMRL demonstrates that aligning training objectives with evaluation metrics through reinforcement learning significantly improves automated grading. While the current model uses a fixed sequence for trait generation, future work may explore dynamic ordering or token-level rewards to further refine the granularity of feedback. Despite the computational overhead of RL, the gains in scoring accuracy and multi-trait consistency mark a significant step forward for automated educational tools.

Summary updated

BibTeX
@inproceedings{do2024autoregressive,
  author = {Heejin Do and Sangwon Ryu and Gary Geunbae Lee},
  title = {{Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.917},
}
Figure