Introduction
Automated Essay Scoring (AES) systems have demonstrated significant proficiency in grading essays when provided with extensive training data for specific prompts. However, they frequently falter in cross-prompt AES scenarios—tasks where a model trained on one topic (e.g., "The benefits of technology") must grade essays on a completely different subject (e.g., "Leadership experiences").
This performance drop stems from the "domain gap": different prompts necessitate distinct vocabularies, genres, and structural approaches. While existing methods attempt to extract shared features like grammar or length, these are often constrained by prompt-specific representation spaces. The authors of PMAES propose that by actively aligning the underlying representations of different prompts, models can more effectively transfer grading knowledge across domains.
Methodology: The PMAES Framework
The proposed PMAES (Prompt-mapping Contrastive Learning for AES) framework addresses the domain gap through a three-part architecture designed to learn prompt-invariant features.
- Shared Encoder: To encourage generalization, the model employs a hierarchical structure using a CNN for sentence-level features and an LSTM for document-level context. Crucially, it utilizes Part-of-Speech (POS) tags instead of raw word embeddings, focusing the model on structural writing patterns rather than topic-specific keywords.
- Scorer Module: This component is designed for flexibility, capable of predicting either a single overall score or multiple attribute-specific scores (e.g., Content, Organization, Grammar). For multi-attribute tasks, an inter-attribute correlation loss is introduced to leverage the inherent relationships between different grading criteria.
- Prompt-mapping Contrastive Learning: This is the core innovation. For a given source essay, the model calculates similarity weights against essays in the target prompt. A learnable "prompt-mapping matrix" then projects the source essay into the target prompt's feature space. A contrastive loss function is applied to pull the original representation and its mapped counterpart closer together, forcing the encoder to ignore prompt-specific "noise."
Experimental Setup
The framework was evaluated using the ASAP++ dataset, an augmented version of the standard Automated Student Assessment Prize dataset. ASAP++ contains 12,978 English essays across eight prompts and includes detailed scores for multiple writing attributes. The authors tested the model across various cross-prompt scenarios, including many-to-one and one-to-one prompt transfers.
Results and Analysis
PMAES demonstrated superior performance over state-of-the-art baselines such as Hi-att, PAES, and CTS. In the single-overall scoring task, PMAES achieved an average Quadratic Weighted Kappa (QWK) of 0.687, surpassing the previous best of 0.658. For multi-attribute scoring, it reached an average QWK of 0.566, leading in nearly all individual categories.
Beyond numerical metrics, the effectiveness of the prompt-mapping approach is evident in the feature space. Using Maximum Mean Discrepancy (MMD), the researchers found that PMAES reduced the representational gap between prompts by approximately 67% compared to standard training methods.
As shown in the t-SNE visualizations, without the mapping mechanism, source and target essays form distinct, isolated clusters. With the application of PMAES, these clusters overlap significantly, indicating that the model has successfully aligned the disparate "languages" of different prompts into a unified representation space.
Conclusion and Future Work
While PMAES significantly advances cross-prompt AES, challenges remain for "source-dependent" prompts that require specific summarization or evidence-citing tasks tied to unique source materials. Future research will focus on extracting even more fine-grained shared writing traits and developing "score-aware" consistency mechanisms to ensure that high-quality essays are mapped to corresponding high-quality regions across different prompt spaces.
Summary updated