Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression

Peter Phandi, Kian Ming Adam Chai, Hwee Tou NgEMNLP 2015 · OpenReview

Abstract: Peter Phandi, Kian Ming A. Chai, Hwee Tou Ng. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

2015.

Introduction

Automated Essay Scoring (AES) systems are designed to reduce the burden of manual grading by providing consistent and immediate feedback. However, a significant bottleneck in their deployment is "prompt specificity." A model trained to grade essays on one topic (the source domain) typically performs poorly when applied to a new topic (the target domain) because it relies heavily on topic-specific vocabulary. Traditionally, overcoming this requires collecting thousands of human-graded essays for every new prompt, a process that is both expensive and time-consuming.

This paper addresses the challenge of domain adaptation: transferring knowledge from a source prompt to a target prompt using only a minimal amount of labeled data. The authors introduce a flexible domain adaptation technique using Correlated Bayesian Linear Ridge Regression (BLRR) that automatically learns the similarity between prompts, enabling effective grading with as few as 10 human-graded examples.

Methodology

The core of the proposed approach is a novel framework based on Correlated Bayesian Linear Ridge Regression. Unlike traditional methods that treat prompts as either identical or entirely independent, this model treats them as two related tasks with a correlation coefficient ($\rho$).

The model utilizes a feature augmentation framework derived from Cholesky decomposition to expand the feature space. This allows the system to distinguish between weights shared across prompts and weights specific to the target prompt. A key innovation is the use of Penalized Maximum Likelihood Estimation to find the optimal $\rho$ for a specific pair of prompts. This flexibility is crucial; for instance, if two prompts are highly similar, $\rho$ approaches 1, whereas for distinct topics, it decreases.

The system utilizes the EASE (Enhanced AI Scoring Engine) feature set, which captures various linguistic dimensions of the essays.

Table 1: Feature types and descriptions used in the AES model.

As shown in Figure 1, the features include length-based metrics, part-of-speech (POS) n-grams to identify grammatical patterns, prompt-specific overlap, and bag-of-words (unigrams and bigrams). The authors noted that lexical features (bag-of-words) often account for the majority of prediction weight—up to 93% in some cases—which explains why simple models fail when the vocabulary shifts between prompts.

Experimental Setup

The researchers evaluated their method using the Automated Student Assessment Prize (ASAP) dataset, a standard benchmark containing eight prompts across three genres: Argumentative (ARG), Response (RES), and Narrative (NAR). The study focused on four adaptation pairs: $1 \to 2$ (ARG), $3 \to 4$ (RES), $5 \to 6$ (RES), and $7 \to 8$ (NAR).

Results and Analysis

The performance was measured using Quadratic Weighted Kappa (QWK), which quantifies the agreement between the automated system and human raters. The proposed method, denoted as ML-$\rho$, was compared against several baselines, including SourceOnly (no adaptation), TargetOnly (trained only on small target data), and EasyAdapt (a method with a fixed correlation).

Table 2: QWK scores for domain adaptation across different pairs and target sample sizes ($n^t$).

As detailed in Figure 5, the ML-$\rho$ method consistently emerged as a robust performer. One of the most significant findings is the "Power of 10": adding just 10 human-graded essays from the target prompt, combined with the domain adaptation framework, significantly outperforms using the source model alone. In the $5 \to 6$ pair with 100 target essays, the model achieved a QWK of 0.713, approaching the in-domain upper bound of 0.775.

The results also highlighted the importance of flexibility. Narrative essays ($7 \to 8$) were found to be much harder to adapt than argumentative ones. The model correctly identified a lower correlation ($\rho \approx 0.63$) for narrative pairs compared to argumentative pairs ($\rho \approx 0.97$). In this difficult narrative case, the proposed method achieved a QWK of 0.621, notably outperforming the TargetOnly model trained on the same amount of data (0.477).

Conclusion

This research demonstrates that flexible domain adaptation via Correlated BLRR can significantly reduce the data requirements for deploying AES systems to new prompts. By automatically learning the relationship between essay topics, the model maintains high accuracy even with very limited target-domain supervision. Future work may explore multi-source adaptation and the use of genre-specific priors to further refine the correlation estimation.

Summary updated

BibTeX
@inproceedings{phandi2015flexible,
  author = {Peter Phandi and Kian Ming Adam Chai and Hwee Tou Ng},
  title = {{Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=7NtHNNGlrd3},
}
Figure