Cross-Prompt Generalization in Automated Essay Scoring

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 240 papers54 conceptsfrozen 2026-10-04

Automated Essay Scoring (AES) systems often suffer from a fundamental "prompt-dependence" trap. When a model is trained to grade essays about The Great Gatsby, it doesn't just learn what good writing looks like; it learns that the word "green light" is a high-value feature. If you then ask that same model to grade a physics essay about thermodynamics, its performance collapses because the vocabulary and themes it memorized are suddenly irrelevant.

The research direction across these 39 papers focuses on cross-prompt generalization: the ability of a model to evaluate writing quality on topics it has never seen before.

The Problem of Topic-Specific Shortcuts

As Shengjie Li 0002 and Vincent Ng 0001 argue in their reflection on the state of the art, many high-performing models are simply "leaderboard-chasing" by exploiting surface-level correlations. If a training set contains mostly long essays with specific keywords, the model learns that "length + keywords = high score."

This creates a "generalization gap." In real-world classrooms, teachers constantly introduce new assignments. Requiring thousands of human-graded examples for every new prompt, as noted by Robert Ridley and Xinyu Dai in their work on Cross-prompt Trait Scoring, is a bottleneck that makes AI grading impractical for everyday use.

Disentangling Content from Quality

To solve this, researchers are moving toward disentangled representation learning. The goal is to force the model to separate what is being said (the content) from how it is being said (the quality).

Zhiwei Jiang and colleagues introduced the PANN (Prompt-Aware Neural Network) framework, which explicitly attempts to strip away prompt-specific signals. Similarly, the Prompt Agnostic Essay Scorer (PAES) by Robert Ridley prioritizes syntactic structures—like Part-of-Speech tags—over raw vocabulary. By looking at the "skeleton" of the writing rather than the "flesh" of the topic, these models maintain accuracy across diverse domains.

Aligning Prompts via Contrastive Learning

Another powerful intuition is that "good writing" should occupy a similar space in a model's mind, regardless of the topic. Yuan Chen and Xia Li developed PMAES, which uses contrastive learning to align representations.

The mechanism works by telling the model: "This high-scoring essay on Prompt A should look more like this high-scoring essay on Prompt B than it looks like a low-scoring essay on its own topic." This forces the neural network to find the universal "latent features" of quality—such as coherence and logical flow—that transcend specific subject matter.

From Holistic Scores to Trait-Specific Rubrics

Modern research is shifting from "holistic" scoring (one single grade) to "trait-specific" scoring (separate grades for grammar, organization, and evidence). This shift actually helps generalization.

Po-Kai Chen proposed MOOSE (Mixture of Ordered Scoring Experts), which mimics human graders by breaking the task into smaller, expert-led decisions. By focusing on specific traits, models can leverage the fact that "Organization" looks similar in a history essay and a science report, even if the "Content" is different. Heejin Do and Gary Geunbae Lee furthered this with ProTACT, which models the natural correlations between traits to improve performance in low-resource settings.

Incorporating Human Cognitive Signals

Perhaps the most novel approach to generalization involves teaching models to "read" like humans. Sandeep Mathias and Pushpak Bhattacharyya explored Cognitively Aided Zero-Shot Grading by training models to predict human gaze patterns.

When humans grade, their eyes linger on complex sentences or jump back to check for coherence. By forcing a model to predict these "cognitive signals," the researchers found the model naturally learned to focus on structural properties of the text rather than just memorizing keywords. This human-centric bias acts as a powerful regularizer, preventing the model from overfitting to the specific vocabulary of the training prompt.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 41 papers
  2. Version 1 · 2026-10-04 · 40 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_prompt_generalization_68fe5be2,
  title = {cross-prompt generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-prompt-generalization/rdr_68fe5be2a74e47a9afcb181ce79b7e0f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_prompt_generalization_in_automated_68fe5be2_v1,
  title = {Cross-Prompt Generalization in Automated Essay Scoring},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-prompt-generalization-in-automated-essay-scoring/rdr_68fe5be2a74e47a9afcb181ce79b7e0f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}