Improving Domain Generalization for Prompt-Aware Essay Scoring via Disentangled Representation Learning

Zhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu, Hua Yu, Zifeng Cheng, Qing Gu

Abstract

Automated Essay Scoring (AES) systems often struggle with domain generalization, where models trained on specific prompts fail to perform on unseen topics. This paper introduces PANN (Prompt-Aware Neural Network), a framework that utilizes disentangled representation learning to separate general writing quality from prompt-specific content. By employing strategies like Contrastive Norm-Angular Alignment (CNAA) and Counterfactual Self-Training (CST), PANN achieves state-of-the-art performance on major benchmarks like ASAP and TOEFL11, demonstrating robust generalization to unseen prompts.

The Challenge of Domain Generalization in AES

Traditional Automated Essay Scoring (AES) systems are typically "prompt-specific," meaning they are trained and evaluated on the same topic. While effective in controlled environments, these models often fail when encountering a new, "unseen" prompt—a scenario known as the domain generalization problem.

Comparison of AES paradigms: Prompt Specific, Prompt Adaptation, and Prompt Generalization.

The fundamental difficulty lies in the entanglement of two distinct features: Essay Quality (general writing traits like grammar and coherence) and Prompt Adherence (relevance to the specific question). Because high-quality essays usually adhere well to prompts, neural models often conflate these signals. When a model overfits to the content of a specific training prompt, it loses the ability to judge the quality of an essay on a different topic.

The PANN Architecture

To address this, the researchers developed the Prompt-Aware Neural Network (PANN). The architecture is designed to explicitly separate quality assessment from content matching through two parallel sub-networks:

  • EQ-net (Essay Quality): A BERT-based encoder focused on extracting features related to the essay's overall writing quality.
  • PA-net (Prompt Adherence): An interaction-based module that directly compares the essay text against the prompt text to measure alignment.
The PANN architecture featuring parallel EQ-net and PA-net modules for disentangled feature extraction.

Disentangled Representation Learning (DRL)

The core innovation of this work is the training framework used to ensure the two sub-networks learn distinct, non-overlapping information.

Contrastive Norm-Angular Alignment (CNAA)

The authors propose a geometric hypothesis: writing quality is reflected in the magnitude (norm) of the feature vector, which should remain stable across different topics. Conversely, content is reflected in the direction (angle) of the vector, which shifts between prompts. By using data augmentation techniques like essay concatenation, the model is forced to learn this magnitude-angle separation.

Counterfactual Self-Training (CST)

To further break the spurious correlation between content and quality, the authors utilize counterfactual reasoning. They ask: "How would the score change if this essay were paired with a different, corrupted prompt?" By training the model on these "counterfactual" pairs, they teach it that an essay's intrinsic quality remains invariant even if its adherence to a specific (corrupted) prompt changes.

Experimental Results

The PANN + DRL framework was evaluated using a "leave-one-prompt-out" cross-validation strategy on two major benchmarks:

  • ASAP (Automated Student Assessment Prize): PANN achieved an average Quadratic Weighted Kappa (QWK) of 0.702, significantly outperforming the standard BERT baseline (0.632).
  • TOEFL11: The model achieved a QWK of 0.695, surpassing the BERT-Dual baseline (0.677).

Ablation studies confirmed that both the Norm-Angular alignment and Counterfactual training are critical components; removing either one led to a significant performance drop, proving the necessity of the disentangled approach.

Conclusion

By separating "how well someone writes" from "what they are writing about," the PANN framework significantly improves the ability of AES systems to generalize to new topics. While the current implementation relies on heuristic-based data augmentation, future work may involve using generative models to create more sophisticated counterfactuals, potentially further enhancing the model's robustness across diverse languages and domains.

Summary updated

BibTeX
@inproceedings{jiang2023improving,
  author = {Zhiwei Jiang and Tianyi Gao and Yafeng Yin and Meng Liu and Hua Yu and Zifeng Cheng and Qing Gu},
  title = {{Improving Domain Generalization for Prompt-Aware Essay Scoring via Disentangled Representation Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.696},
}
Figure