Modeling Why Essay Graders Disagree

When two trained teachers grade the same essay, they often give different scores. In developmental-education placement, only 55% of double-rated essays received the same skill classification (Enhancing Writing Proficiency Classification in Developmental Education). Expert agreement on spoken fluency was 53.5% (Deshmukh et al.). Automated graders built on large language models, and "LLM-as-a-judge" systems that use such a model as the grader, mostly treat this spread as noise and predict one number. Recent work predicts the whole spread of human scores instead. That tells a teacher that graders would split, but not why.

We propose a grader that predicts the distribution of human scores and explains it in a form a teacher can check. The explanation has two parts: which passages of the essay carry the contested evidence, and which grading priorities pull raters in different directions. The central bet is that much disagreement is structured. Raters see roughly the same strengths and weaknesses but weight them differently, for example "strong ideas, weak mechanics." If that is right, disagreement can be predicted from two things: trait evidence located in specific passages, and a small set of learned rater weighting profiles.

Core Claim

Break the essay into discourse segments, meaning paragraph-level units. Give each segment a header carrying the prompt, the thesis and a one-line summary of what came before. Have specialist agents extract quoted evidence and a trait score per segment. Then pass the resulting trait-evidence vector through a small latent-class model of rater weighting profiles. We claim this pipeline predicts the spread of human scores as well as a fine-tuned distributional judge. It also does three things that judge cannot: it predicts individual raters' held-out scores better once their profile is inferred, it localizes disagreement to passages that humans themselves flag, and it reduces the tendency of LLM graders to over-penalize surface errors.

Concretely, let $t(x)$ be the vector of trait scores the agents assign to essay $x$. The model has $K$ rater profiles. Each profile $k$ has trait weights $w_k$ and ordinal cutpoints $c_k$, and $\pi_k$ is the share of raters in that profile. The predicted distribution over holistic score $y$ is

$$P(y \mid x) = \sum_{k=1}^{K} \pi_k \, \mathrm{OrdLogit}\big(y \mid w_k^\top t(x),\, c_k\big).$$

For a specific rater $r$, we replace $\pi_k$ with a posterior over profiles computed from that rater's past scores. The mixture form is standard latent-class modeling of annotators. We use it because it turns a split distribution into a readable statement: "profile A weights Idea Development heavily, profile B weights Mechanical Conventions, and this essay scores high on one and low on the other in paragraphs 2–3."

Relation to Prior Work

Two papers motivate the gap. Distributional LLM-as-a-Judge fine-tunes a single judge to match human label distributions, using a divergence-plus-cross-entropy loss with adversarial perturbations. It reports a KL divergence of 0.23 on SNLI, versus 0.60 for single-point training. Its rationale describes only one sampled judgment, so the spread goes unexplained. We keep its target and its loss family, but move the learned part into an interpretable aggregator over agent-extracted evidence.

Multi-Agent-as-Judge builds stakeholder personas from domain literature and has them debate before a synthesizer collapses their views into one score. It reaches Spearman 0.47 on StorySparkQA, versus 0.31 for G-Eval. Its personas are invented rather than fit to real raters, and the collapse discards the spread. Our profiles are estimated from actual rater scores, and we never collapse.

Three papers supply method. MTS decomposes essay quality into traits and makes the model quote evidence before scoring each one. It reaches QWK 0.560 on ASAP with Llama2-13b, versus 0.407 for vanilla ChatGPT. We reuse quote-then-score, but apply it per segment. CAFES diagnoses a "Grammar Police" bias, in which LLM graders over-penalize mechanical errors. It counters the bias with a positive-evidence pool agent and reports a 21% relative QWK gain on EssayJudge. We use that diagnosis as a measurable outcome. Zhang et al. prepend a global header to every chunk of a long document. InstructCoder uses fixed windows with 50% overlap. The first gives our segmentation design; the second is the comparison condition.

Two papers constrain the design. Single-Agent vs. Multi-Agent LLM Strategies found that a single agent with few-shot examples reached 82.8% exact match. That beat the multi-agent decision tree, which suffered coordination problems once examples were split across agents. We respond by keeping agents zero-shot and narrow: they only extract and score evidence. All calibration to human raters happens in the small statistical aggregator, so no examples are passed between agents. Da Corte and Baptista found that Idea Development, Critical Thinking and Organization were the most predictive criteria and Mechanical Conventions the least. Their data also records six criterion scores per rater, which is the structure our model needs.

Research Program

The pipeline has three stages. First, a segmenter splits each essay at paragraph and function boundaries: introduction, body arguments, conclusion. It attaches the persistent header described above to each segment. Second, trait agents score each segment. There is one agent per rubric trait, and the agents are deliberately diverse: different base models where possible, to reduce the shared blind spots that arise when all agents share the same weights. Each agent must quote evidence before scoring. Global traits such as Organization are scored by a reconciler agent. It sees the whole essay plus the segment evidence, so a pile of local errors is weighed against the macro-level argument. Third, the latent-profile aggregator is fit on multi-rater data. We choose $K$ by held-out likelihood.

Explanations are produced from the fitted model's own quantities, not from free text. They name the profiles whose predictions diverge, the traits responsible, and the segments whose evidence drives those traits.

Multi-rater data with per-trait scores is scarce, so the program uses several datasets. SummEval has multiple raters across four quality dimensions, and Distributional LLM-as-a-Judge already used it distributionally. The TCC placement set gives six criteria per rater on 49 double-rated essays (Da Corte and Baptista). The Kyushu reflection set reports Krippendorff's alpha of 0.8386, which implies multiple raters (Li et al.). CLASSE provides content and wording scores, with wording rated only moderately reliably (CLASSE).

We will also collect one new resource. It covers about 300 essays drawn from ASAP and TOEFL11, each scored by five raters. Every rater gives per-trait scores and highlights the passages that most moved their holistic score. These highlights are the ground truth for disagreement localization, which no existing dataset in the file provides.

Baselines

The naive reference is a single frontier model prompted once with the whole essay and rubric. We read a distribution from its score-token probabilities. The file's anchor for how weak this can be is vanilla ChatGPT's QWK of 0.407 on ASAP (MTS).

The strongest existing method for the distribution target is Distributional LLM-as-a-Judge. We will fine-tune it on the same open base model, the same training splits and the same rater distributions, with an equal hyperparameter search budget. Its reported figure is KL 0.23 on SNLI. The file gives no SummEval number, so no reported result was found on Lacuna for its SummEval KL, and we will reproduce it ourselves. Beating it means lower held-out KL or Jensen–Shannon divergence. Matching it within confidence intervals while also winning on the rater-specific and localization tests counts as success for the explanatory claim.

The strongest method for majority-score accuracy is MTS, run as reported. Its anchor is QWK 0.560 on ASAP with Llama2-13b. We also run it on the same stronger base model we use, so model size is not a confound. For the multi-agent comparison, we run MAJ-EVAL's persona debate (anchor: Spearman 0.47 on StorySparkQA). We keep each persona's score separate rather than collapsing them, which gives it the fairest chance to produce a distribution. We also run CAFES-style scoring. Its relative QWK gain of 21% on EssayJudge is the only reported number, and the file gives no raw trait values.

Li et al.'s single-agent few-shot prompt (82.8% exact match on Kyushu) is run with the same exemplars our aggregator sees. Every system gets the same number of labeled essays and a matched budget of LLM calls.

Four ablations isolate the key ingredients. The first sets $K=1$, a single population weighting, which removes the rater profiles. The second replaces discourse segments with fixed 50%-overlap windows. The third removes segmentation entirely and has the agents score the whole essay. The fourth removes the global reconciler.

Why not just…?

Why not just sample one judge many times, or read its score probabilities, to get a distribution? This is cheap and a real threat. However, sampling spread measures the model's own uncertainty, which need not match the structure of human disagreement. It also cannot say which rater would give which score. The token-probability baseline tests this directly. If it matches our held-out KL and our per-rater likelihood, the profile model adds nothing and we will say so.

Why not just fine-tune the distributional judge and stop? On distribution-matching alone it may win, given its SNLI result. Our claim does not rest on beating it on KL. It rests on three tests the fine-tuned judge cannot take in the same form. It has no rater profile to infer, so it cannot predict a specific rater's held-out scores. It gives no span-level account of disagreement to compare against human highlights. And the explanations it would produce for the grammar-bias test would describe only a single sampled judgment. If it ties us on KL, the contribution is interpretability at equal accuracy. If it beats us clearly, the structured-disagreement hypothesis is weakened.

Why not just give a long-context model the whole essay, since essays fit in context anyway? For ASAP- and TOEFL-length essays this objection is largely correct on context grounds. We therefore do not justify segmentation by length. We justify it by where evidence sits, and the no-segmentation ablation settles the question. If whole-essay agents localize disagreement and resist grammar bias equally well, we drop segmentation for short texts. We would then keep it only for the longer TCC essays and the fixed-window comparison.

Why not just use MAJ-EVAL's personas without collapsing them? Personas drawn from literature are plausible but unfit. The uncollapsed-persona baseline shows whether invented perspectives recover the real split as well as profiles estimated from rater data.

Proposed Experiments and Decisive Checks

The primary metric is held-out divergence between predicted and empirical rater distributions, on SummEval, CLASSE, Kyushu and the new corpus. Some sets have only two raters, where a two-rater "distribution" is coarse. There we report the log-likelihood of each held-out rater's score, plus the area under the curve for predicting whether the two raters differ by more than one point. Majority-score QWK is reported throughout, so that no gain in distribution quality hides a loss in accuracy.

The first decisive check targets the rater-profile hypothesis. We infer each rater's profile from 20 of their essays and predict their remaining scores. The hypothesis fails if this does no better than population-level prediction.

The second decisive check targets localization, on the new corpus. We rank segments by how much they contribute to divergence between profiles, and measure overlap with the passages raters highlighted. The comparison is the fixed-window condition and a random-segment control.

The third check targets grammar bias. We build matched pairs of essays: the same content, with mechanical errors injected or removed. We measure how far each system's predicted score shifts, relative to how far human raters' scores shift on a rated subset. A system that penalizes injected errors more than humans do is judged to over-penalize them.

If the first check fails, we abandon the claim that disagreement is structured. We would then report only whether segment-level evidence helps as a distributional judge.