Abstract
Automated Essay Scoring (AES) systems often struggle to generalize to unseen prompts, frequently overfitting to topic-specific vocabulary. This paper introduces GAPS (Grammar-aware cross-prompt trait scoring), a framework that leverages Grammar Error Correction (GEC) to extract universal writing features. By comparing original essays with their corrected versions through a dual-stream encoder and a knowledge-sharing mechanism, GAPS significantly improves performance on prompt-agnostic traits like Conventions and Sentence Fluency. Evaluations on the ASAP and ASAP++ datasets demonstrate that GAPS outperforms existing state-of-the-art models in cross-prompt scenarios.
Introduction
Automated Essay Scoring (AES) provides a scalable solution for educational assessment, yet it faces a significant "generalization" hurdle. Most models are trained on specific prompts and fail when applied to new topics because they overfit to prompt-specific vocabulary. This is particularly problematic for "prompt-agnostic" traits—such as Conventions (spelling, punctuation) and Sentence Fluency—which should ideally be evaluated consistently regardless of the subject matter. Current state-of-the-art models exhibit a performance gap of up to 20% between content-related and fundamental grammar traits in cross-prompt settings.
Methodology: The GAPS Framework
The authors propose GAPS, a two-step method designed to isolate syntactic quality from semantic content.
- Essay Correction & Tagging: Essays are processed through a T5-based Grammar Error Correction (GEC) model. Instead of using only the "clean" output, a Grammar Correction Tagging (GCT) system explicitly marks edits (e.g.,
<corr> R: helps </corr>), providing the model with direct signals about errors. - Dual-Stream Encoding: The system employs two identical encoders for the original and corrected versions of the essay.
- Knowledge-Sharing Layer: A cross-attention mechanism allows the model to "trade knowledge" between representations. The corrected essay serves as a "gold standard," helping the model distinguish between intended meaning and syntactic errors.
- Trait-Specific Scoring: The model uses a hierarchical structure to score individual traits (e.g., Organization, Word Choice, Content) based on both sentence and document-level features.
Experimental Setup
The study utilizes the ASAP and ASAP++ datasets, which include eight distinct prompts covering argumentative and narrative styles. To evaluate generalization, the authors employed a "leave-one-prompt-out" cross-validation approach, ensuring the model was tested on truly unseen topics.
Key Results and Insights
GAPS was evaluated using the Quadratic Weighted Kappa (QWK) metric. The results indicate that grammar acts as a "universal anchor," providing a stable signal across varying topics.
- Superior Generalization: GAPS consistently outperformed previous state-of-the-art models, such as ProTACT, in cross-prompt scenarios.
- Closing the Trait Gap: The most significant improvements were observed in the Conventions trait. Notably, in "Prompt 7"—a narrative prompt with a unique structure—GAPS showed remarkable robustness.
- Semantic Benefits: Interestingly, providing grammar error data also improved scores for Content and Narrativity. By "filtering" grammatical noise, the model more effectively captured the underlying semantic context.
- Ablation Analysis: Removing the Knowledge Sharing (KS) component led to a sharp decline in performance for Word Choice and Sentence Fluency, confirming the importance of the interaction between original and corrected texts.
Limitations and Future Work
The primary bottleneck for GAPS is its GEC Dependency. The scoring accuracy is tied to the quality of the underlying Grammar Error Correction model. Future work will explore more advanced GEC models, application to diverse writing genres (e.g., technical or creative writing), and the development of actionable feedback mechanisms for students based on recurring error types.
Summary updated