Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions

Mengxue Zhang, Neil T. Heffernan, Andrew S. Lan

Abstract

Automated Scoring (AS) systems typically treat human-provided grades as an objective ground truth. However, human grading is inherently subjective, leading to "noisy labels" that can degrade model performance. This paper proposes a framework to model individual scorer preferences—termed "scorer fingerprints"—by integrating scorer-specific bias and variance parameters into a BERT-based classification model. Using a dataset of over 118,000 short-answer math responses, the authors demonstrate that accounting for these systematic biases improves scoring accuracy (Kappa) by over 0.02 and reveals distinct archetypes of grading behavior.

Introduction

The development of Automated Scoring (AS) systems for open-ended questions has traditionally relied on supervised learning, where models are trained to replicate scores provided by human experts. The efficacy of these systems is often limited by the assumption that human scores are consistent and objective. In reality, grading is a subjective process. In complex domains such as mathematics, inter-scorer agreement can be remarkably low; previous research has noted Kappa scores as low as 0.083 between different human scorers.

When models are trained on such inconsistent data, they inherit the noise of the labels. Most existing AS models ignore the identity of the scorer, effectively treating systematic human bias as random error. This paper argues that human variability is not merely noise but a predictable feature that can be modeled to improve the precision and reliability of automated grading.

Proposed Methodology

The researchers introduced a framework that incorporates scorer-specific information directly into the classification component of an AS system. The architecture is built upon three primary pillars:

1. Representation and Scorer Parameters

The system utilizes a pre-trained BERT model to generate high-dimensional embeddings for question-response pairs. To account for individual differences, the authors introduce two scalar parameters for every scorer:

  • Bias ($b$): An offset for each score category that captures whether a scorer is generally strict or lenient.
  • Variance/Temperature ($\alpha$): A parameter that controls the "certainty" or consistency of a scorer's decisions.

2. Content-Driven Modeling

While a flat bias parameter is useful, the authors found that scorer tendencies often fluctuate based on the specific content of a student's answer. Their most advanced model makes the bias and variance parameters dependent on the response embeddings. This allows the model to capture nuanced behaviors, such as a teacher who is lenient regarding minor calculation errors but highly strict regarding the logical formatting of a proof.

3. Ordinal Log Loss (OLL)

Standard classification models often use cross-entropy loss, which treats all misclassifications equally. However, math scores (typically 0–4) are ordinal. A model that predicts a "4" when the true score is "0" has made a much more significant error than one predicting a "1" for a "0." By employing Ordinal Log Loss (OLL), the researchers penalize distant misclassifications more heavily, better aligning the model with the mathematical nature of the grading scale.

Experimental Setup

The study utilized a real-world dataset from an online learning platform. The initial dataset comprised 141,612 responses to 2,042 questions, graded by 891 teachers. To ensure statistical significance for individual scorer modeling, the researchers filtered the data to include only those who had graded at least 100 responses. This resulted in a final dataset of 118,079 responses from 203 scorers, with scores ranging from 0 (no credit) to 4 (full credit).

Results and Scorer Archetypes

The inclusion of scorer-specific parameters yielded a significant performance boost. The best-performing configuration (Content-driven + OLL) achieved a Kappa score of 0.641, compared to the baseline's 0.614. Furthermore, the use of OLL consistently outperformed Cross-Entropy and Mean Squared Error across all metrics, including AUC and RMSE.

Beyond performance metrics, the model allowed for a fascinating analysis of human behavior. By applying Expectation-Maximization clustering to the learned embeddings, the authors identified six distinct "scorer archetypes":

  • The Strict Grader: Exhibits a strong bias toward the lowest possible scores.
  • The Lenient Grader: Shows a consistent preference for higher scores.
  • The Polarizer: Adopts an "all or nothing" approach, frequently awarding 0s or 4s while avoiding middle-ground scores.
  • The Conservative Grader: Tends to stay in the middle of the scale, rarely awarding the highest possible marks.
  • The Reliable Grader: Found in nearly half of the scorers (Cluster 4), these individuals are relatively unbiased, with their scores depending primarily on the objective quality of the student's response.

Discussion and Future Work

This research demonstrates that scorer subjectivity is systematic rather than random. By modeling these "fingerprints," AS systems can better navigate the noise inherent in human-labeled datasets.

However, limitations remain. The current dataset lacked demographic metadata (e.g., gender, location, or teacher-student relationships), which are essential for a deeper analysis of fairness and the root causes of bias. Additionally, while the model is excellent at predicting what a specific teacher would grade, a truly "fair" system may eventually need to use these learned embeddings to "de-bias" or normalize scores across an entire platform. Future work will explore whether these scorer-preference layers can be integrated into other architectures beyond BERT to create more robust and equitable educational tools.

Summary updated

BibTeX
@inproceedings{zhang2023modeling,
  author = {Mengxue Zhang and Neil T. Heffernan and Andrew S. Lan},
  title = {{Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions}},
  booktitle = {Edm},
  year = {2023},
}
Figure