Cross-Prompt Generalization in Automated Essay Scoring

53 concepts42 papers1 proposal

Automated Essay Scoring (AES) systems have reached a point where they can match human raters when trained on a specific topic. However, these models often suffer from a "generalization gap": a significant performance decline when applied to new essay prompts or subject domains not present in their training data. As Li et al. and Li et al. observe, this lack of robustness suggests that models are often learning prompt-specific shortcuts—such as vocabulary distributions or surface-level correlations like essay length—rather than universal principles of writing quality.

The Content-Structure Conflict

The primary bottleneck in cross-prompt generalization is the entanglement of content and quality. When a model is trained on a single prompt, it learns that specific keywords are indicators of high scores. However, these lexical features are brittle; a model trained to recognize quality in an essay about "recycling" will fail when evaluating an essay about "space exploration" because the relevant vocabulary has shifted entirely.

To address this, researchers have explored ways to decouple writing quality from specific subject matter:

  • Structural Proxies: Beigman Klebanov et al. argue that the structure of an argument—how claims and premises are linked—serves as a robust, prompt-independent proxy for quality. By focusing on the "how" rather than the "what," models can maintain accuracy across diverse topics.
  • Linguistic Feature Engineering: Surprisingly, complex neural networks are not always the best solution for generalization. Cozma et al. and Yang et al. demonstrate that handcrafted linguistic features and string kernels can outperform deep learning architectures in cross-prompt scenarios, as these features capture fundamental writing characteristics that transcend specific themes.
  • Hybrid Semantic Modeling: In short-answer grading, Levy et al. show that balancing lexicalized features (keywords) with non-lexicalized semantic features (textual entailment) allows systems to evaluate the underlying meaning of a response even in unfamiliar domains.

Disentangling Traits from Topics

Modern approaches attempt to solve the generalization problem by explicitly separating prompt-specific nuances from general writing traits. Ridley et al. introduced the Cross-prompt Trait Scorer (CTS), which uses a "trait-attention" mechanism to grade specific aspects like organization or word choice across unseen topics. This allows the model to identify universal indicators of "good organization" that apply regardless of the essay's subject.

Building on this, Han et al. proposed a multi-level disentanglement framework. Their intuition is that a trait like "Content" means something different for every prompt, but the relationship between content and score follows a learnable pattern. By disentangling these levels, the model can adapt to the linguistic variety of new educational settings without requiring massive amounts of new labeled data.

Overcoming the Cold Start Problem

A major practical hurdle is the "cold start" problem: grading essays for a brand-new prompt with zero human-labeled examples. Jin et al. addressed this using a two-stage deep neural network. The system first uses a shallow model to identify "extreme" quality essays (very high or very low) to serve as pseudo-training data. A more sophisticated network is then trained on these extremes, effectively bootstrapping its understanding of quality for the new prompt without human intervention.

Similarly, in the realm of feedback, Magooda et al. and Filighera et al. emphasize that true generalization requires moving beyond a single holistic score. By providing explainable, trait-specific feedback that is not tightly coupled to a single source text, automated systems become more viable for real-world, dynamic classroom environments.

Key Foundational Work

ENLP Tutorial Notes -- Slides (2008) — contributed 1 concept elements to this direction.
Automated Scoring Using A Hybrid Feature Identification Technique (1998) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 41 papers
  2. Version 1 · 2026-10-04 · 40 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_prompt_generalization_68fe5be2,
  title = {cross-prompt generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-prompt-generalization/rdr_68fe5be2a74e47a9afcb181ce79b7e0f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_cross_prompt_generalization_68fe5be2_v2,
  title = {cross-prompt generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-prompt-generalization/rdr_68fe5be2a74e47a9afcb181ce79b7e0f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake