Towards Prompt Generalization: Grammar-aware Cross-Prompt Automated Essay Scoring

Heejin Do, Taehee Park, Sangwon Ryu, Gary LeeNAACL 2025 · DOI

Abstract

Automated Essay Scoring (AES) systems often struggle to generalize to unseen prompts, frequently overfitting to topic-specific vocabulary. This paper introduces GAPS (Grammar-aware cross-prompt trait scoring), a framework that leverages Grammar Error Correction (GEC) to extract universal writing features. By comparing original essays with their corrected versions through a dual-stream encoder and a knowledge-sharing mechanism, GAPS significantly improves performance on prompt-agnostic traits like Conventions and Sentence Fluency. Evaluations on the ASAP and ASAP++ datasets demonstrate that GAPS outperforms existing state-of-the-art models in cross-prompt scenarios.

Introduction

Automated Essay Scoring (AES) provides a scalable solution for educational assessment, yet it faces a significant "generalization" hurdle. Most models are trained on specific prompts and fail when applied to new topics because they overfit to prompt-specific vocabulary. This is particularly problematic for "prompt-agnostic" traits—such as Conventions (spelling, punctuation) and Sentence Fluency—which should ideally be evaluated consistently regardless of the subject matter. Current state-of-the-art models exhibit a performance gap of up to 20% between content-related and fundamental grammar traits in cross-prompt settings.

Methodology: The GAPS Framework

The authors propose GAPS, a two-step method designed to isolate syntactic quality from semantic content.

The GAPS architecture utilizes a dual-stream encoder to compare original and corrected essays, facilitating knowledge sharing for trait-specific scoring.
  1. Essay Correction & Tagging: Essays are processed through a T5-based Grammar Error Correction (GEC) model. Instead of using only the "clean" output, a Grammar Correction Tagging (GCT) system explicitly marks edits (e.g., <corr> R: helps </corr>), providing the model with direct signals about errors.
  2. Dual-Stream Encoding: The system employs two identical encoders for the original and corrected versions of the essay.
  3. Knowledge-Sharing Layer: A cross-attention mechanism allows the model to "trade knowledge" between representations. The corrected essay serves as a "gold standard," helping the model distinguish between intended meaning and syntactic errors.
  4. Trait-Specific Scoring: The model uses a hierarchical structure to score individual traits (e.g., Organization, Word Choice, Content) based on both sentence and document-level features.

Experimental Setup

The study utilizes the ASAP and ASAP++ datasets, which include eight distinct prompts covering argumentative and narrative styles. To evaluate generalization, the authors employed a "leave-one-prompt-out" cross-validation approach, ensuring the model was tested on truly unseen topics.

Key Results and Insights

GAPS was evaluated using the Quadratic Weighted Kappa (QWK) metric. The results indicate that grammar acts as a "universal anchor," providing a stable signal across varying topics.

Performance comparison of GAPS against state-of-the-art models across various writing traits.
  • Superior Generalization: GAPS consistently outperformed previous state-of-the-art models, such as ProTACT, in cross-prompt scenarios.
  • Closing the Trait Gap: The most significant improvements were observed in the Conventions trait. Notably, in "Prompt 7"—a narrative prompt with a unique structure—GAPS showed remarkable robustness.
  • Semantic Benefits: Interestingly, providing grammar error data also improved scores for Content and Narrativity. By "filtering" grammatical noise, the model more effectively captured the underlying semantic context.
  • Ablation Analysis: Removing the Knowledge Sharing (KS) component led to a sharp decline in performance for Word Choice and Sentence Fluency, confirming the importance of the interaction between original and corrected texts.

Limitations and Future Work

The primary bottleneck for GAPS is its GEC Dependency. The scoring accuracy is tied to the quality of the underlying Grammar Error Correction model. Future work will explore more advanced GEC models, application to diverse writing genres (e.g., technical or creative writing), and the development of actionable feedback mechanisms for students based on recurring error types.

Summary updated

BibTeX
@inproceedings{do2025towards,
  author = {Heejin Do and Taehee Park and Sangwon Ryu and Gary Lee},
  title = {{Towards Prompt Generalization: Grammar-aware Cross-Prompt Automated Essay Scoring}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.153},
}
Figure