Introduction
Automated Essay Scoring (AES) has traditionally focused on providing a single "holistic" grade. However, for educational purposes, students require nuanced feedback across multiple dimensions, such as organization, fluency, and grammar. This paper, presented at EMNLP 2024, introduces SaMRL (Scoring-aware Multi-reward Reinforcement Learning), a framework designed to provide high-quality multi-trait feedback by leveraging autoregressive models and reinforcement learning.
The Challenge: Metric Alignment and Model Scalability
The development of effective multi-trait AES systems faces two primary hurdles:
- The Metric Gap: The standard evaluation metric for AES is the Quadratic Weighted Kappa (QWK). Because QWK is non-differentiable, most neural models are trained using Mean Squared Error (MSE). This misalignment often leads to models that perform well on loss functions but fail to capture the grading nuances prioritized by human evaluators.
- Model Limitations: Traditional classification or regression models struggle with wide score ranges (e.g., 0–60) and often require separate output "heads" for each trait, which is computationally inefficient and ignores the dependencies between different grading criteria.
The SaMRL Framework
To address these issues, the authors propose treating essay scoring as a sequence generation task. Using a T5-based architecture, the model "writes" scores as a sequence of tokens (e.g., [Trait1 4, Trait2 3...]). This autoregressive approach allows a single model to capture all traits while modeling the correlations between them.
The core innovation is the Scoring-aware Multi-reward RL mechanism. Since QWK cannot be used as a direct loss function, it is employed as a reward within a Policy Gradient (PPO) framework. The system incorporates:
- Bi-directional QWK Rewards: Rewards are calculated both across batches (comparing different essays) and across traits (ensuring internal consistency within a single essay).
- Multi-objective Optimization: The reward function combines QWK with an MSE penalty for numerical precision and a KL-divergence term (using a frozen "anchor" model) to maintain output formatting and prevent model drift.
Experimental Results
The researchers evaluated SaMRL on the ASAP, ASAP++, and Feedback Prize benchmarks. SaMRL consistently achieved state-of-the-art performance, particularly on complex argumentative and narrative essays.
Key findings include:
- Superior Accuracy: On the ASAP++ dataset, SaMRL-large achieved an average QWK of 0.705, outperforming previous autoregressive baselines like ArTS.
- Robustness: The model showed significant improvements on prompts with wide score ranges (such as P7 and P8), where traditional classification models typically degrade due to the high number of possible classes.
- Trait Correlation: By generating scores sequentially, the model effectively learned how different traits (like sentence fluency and conventions) relate to one another, capturing dependencies that independent scoring heads miss.
Conclusion and Future Work
SaMRL demonstrates that aligning training objectives with evaluation metrics through reinforcement learning significantly improves automated grading. While the current model uses a fixed sequence for trait generation, future work may explore dynamic ordering or token-level rewards to further refine the granularity of feedback. Despite the computational overhead of RL, the gains in scoring accuracy and multi-trait consistency mark a significant step forward for automated educational tools.
Summary updated