Automated essay scorers are trained on essays written for some prompts and then used on essays for a new prompt that has no human-scored examples. This is called cross-prompt scoring. Teachers want more than one overall grade, so it is natural to first predict sub-scores for writing traits (organization, word choice, conventions) and then combine them into the overall, or holistic, score. Yet the strongest recent study found the reverse of what one would hope. Feeding predicted trait scores into the holistic scorer lowered average agreement with human raters on ASAP. Agreement is measured by quadratic weighted kappa (QWK), a chance-corrected statistic that penalizes large score disagreements more than small ones; it fell from 0.698 to 0.683 (Conundrums in Cross-Prompt Automated Essay Scoring). The authors blamed error propagation, meaning upstream mistakes passed downstream as if they were facts, and dropped traits. We think this diagnosis is incomplete, and that the fix is cheap.
Our hypothesis is that trait predictions on an unseen prompt are not randomly noisy. They are biased in a prompt-specific way: some traits shift up or down together for that topic. The holistic scorer fails because it was trained on trait inputs whose errors look nothing like the errors it meets on the new prompt. If this is right, there is a simple remedy. Train the holistic scorer only on trait predictions made under the same held-out-prompt condition it will face at test time ("shift-matched stacking"). Pass trait probability distributions instead of single hard labels. Then, on the new prompt, estimate without any labels how much each trait signal can be trusted, and turn its influence down when it looks unreliable. The proposal aims to show that traits can improve holistic scoring on unseen prompts, and to measure exactly how much of the "trait conundrum" each of these three repairs removes.
Core Claim
Our central claim is that the degradation is mostly a train/test mismatch in the distribution of intermediate errors, not an inherent cost of decomposing the score. This claim is testable and can be falsified with one measurement. On ASAP++ we know the true human trait scores for every target prompt. We can therefore compute a propagation gap: holistic QWK when the scorer receives gold traits, minus holistic QWK when it receives predicted traits. We can then track how this gap changes under each repair.
If shift-matched stacking plus soft inputs closes most of the gap and lifts the trait-informed scorer above the trait-free 0.698, the conundrum is a training artifact. If the gap persists even with perfectly matched error distributions, trait errors on new prompts are too large to be useful. Li and Ng's decision to abandon traits would then be vindicated with a mechanism attached. Either outcome is informative.
The hypothesis also explains an apparent contradiction in the literature. On ICLE++, adding traits generally helped cross-prompt holistic scoring (ICLE++), while on ASAP it hurt (Conundrums). Under our account, the sign depends on how closely the trait errors seen when training the holistic scorer resemble those seen at test. This is something we can measure on both corpora.
Relation to Prior Work
The motivating evidence comes from two papers. Conundrums shows a feature-based multi-layer perceptron beating neural transfer systems, and shows trait-informed scoring falling below it; it identifies propagation but does not try to repair it. TRATES breaks the scoring rubric into diagnostic questions answered by a large language model (LLM) as High/Medium/Low, then feeds those answers to a shallow regressor. That hard three-way commitment is exactly where errors become frozen into the feature table. We reuse TRATES's question decomposition, but keep the LLM's probabilities over the three answers instead of only its top choice.
The method templates come from outside essay scoring. BioMLN turned the confidence margins of a support vector machine into soft logical constraints rather than discrete trigger labels, and gained 10 F1 points on complex events. Claim-Dissector treats evidence relevance as a hidden variable that is optimized jointly with the final verdict. Both remove hard commitments, which we borrow. Neither faces our specific problem: a domain shift that changes the reliability of the upstream stage between training and test. Shift-matched training and label-free reliability gating are aimed at that problem.
Several papers set constraints on the design. Parallel multi-trait models such as CTS and SICTS sidestep sequential propagation by predicting all traits at once, but they give up an explicit trait-to-holistic step. Relation-aware models, ProTACT and GAT-AES, show that traits are strongly correlated with one another (pairwise Pearson correlation of at least 0.6). That correlation is why trait errors on one prompt can move together, which is the structure our account predicts. Omethi handles cascades by abstaining and passing unconfident items to the next stage, ending with humans. We adapt the abstention idea to the feature level, down-weighting a trait rather than rejecting an essay, because cross-prompt scoring has no human fallback.
Research Program
The first stage is diagnosis. We reproduce the trait-then-holistic pipeline of Conundrums on ASAP/ASAP++ with leave-one-prompt-out evaluation, so that each of the eight prompts is held out in turn. For each held-out prompt we compute the propagation gap. We then break trait error into two parts: a per-prompt offset (the mean error across that prompt's essays) and the remaining essay-level scatter. Our hypothesis predicts that the offset accounts for a large share of the damage, and that it is shared across correlated traits.
The second stage builds the repaired pipeline in three separable pieces.
Shift-matched stacking means training the trait predictors with nested leave-one-prompt-out. The trait inputs the holistic scorer sees during training for source prompt $p$ come from trait models that never saw prompt $p$. This mirrors the test condition. It is cross-fitting, a standard technique for stacked models, applied at the level of prompts rather than random folds. We expect the prompt level to be what matters.
Soft intermediates means passing trait predictions as full distributions. For a regression trait model this is its predicted mean and variance. For the TRATES-style LLM questions it is the normalized token probabilities over High/Medium/Low.
Reliability gating means scaling each trait's contribution on the target prompt by a label-free trust estimate, $r_{t,p}$. We use two independent upstream estimators for each trait: the feature model and the LLM rubric answers. We set $r_{t,p}$ to their rank correlation across the target prompt's essays, which requires no labels. The holistic prediction for essay $x$ is
$$\hat{y}(x) = g\big(f(x),\; r_{1,p}\,\tilde z_1(x), \dots, r_{T,p}\,\tilde z_T(x),\; r_{1,p},\dots,r_{T,p}\big).$$
Here $f(x)$ is the Li and Ng feature vector, $\tilde z_t(x)$ is the soft prediction for trait $t$, $T$ is the number of traits, and $g$ is the downstream multi-layer perceptron. The $r$ values are also supplied as inputs so that $g$ can learn how much to trust traits from the held-out-prompt folds, where the true reliability is known.
The third stage tests transfer across learner populations. We train on ASAP++ and evaluate on ICLE++ and on ELLIPSE's 44 prompts. ELLIPSE's many prompts give enough data points to check whether the gate's $r_{t,p}$ actually predicts per-prompt trait QWK.
Baselines
The naive reference point is a length-only scorer, since length dominates cross-prompt holistic prediction (Argumentation reports QWK .518 for length alone on TOEFL). We rerun it on ASAP under our splits.
The strongest existing method is the "All Feats Filtered" multi-layer perceptron from Conundrums. It reports an average holistic QWK of 0.698 across the eight ASAP prompts, ahead of PMAES at 0.687. It is the strongest competitor because it is the trait-free scorer our method must beat for traits to be worth including. We use the same feature set, the same correlation filter, and the same leave-one-prompt-out protocol. Its tuning budget is matched to the downstream scorer $g$ in our pipeline.
The direct competitor is the paper's own hard trait pipeline, reported at 0.683. ProTACT with traits, reported at 0.674 (Conundrums), serves as the neural trait-aware reference.
We also include two stronger alternatives. The first is a TRATES-style regressor on hard ternary answers; no reported holistic QWK for TRATES was found on Lacuna, so we run it ourselves. The second is a parallel multi-task model in the style of SICTS, whose cross-prompt holistic number is likewise not reported in the file.
Ablations add shift-matched stacking alone, soft inputs alone, and gating alone, and also include an oracle using gold traits. All systems use the same essays, the same frozen LLM, and five random seeds. We report the mean and the standard deviation across prompts.
Beating the strongest baseline means exceeding 0.698 average QWK by more than the across-seed spread. Separately, the propagation gap must shrink. An improvement in the gap alone is not enough.
Why not just…?
Why not just drop traits, as Li and Ng did? This is the default, and it may remain best for the holistic score alone. It cannot provide the trait feedback learners need (CTS). It also relies heavily on length. The comparison against 0.698 settles the question directly. If no repaired variant beats it, we report that traits do not help holistic scoring, but we can then say why, using the gap decomposition.
Why not just do ordinary stacking with cross-validation? This is the real threat. Shift-matched stacking is cross-fitting, and it may be the entire effect. Our ablation separates random-fold cross-fitting from prompt-held-out cross-fitting. If random folds close the gap just as well, the problem was ordinary overfitting rather than prompt shift. That would weaken our account, and we would say so.
Why not just use a parallel multi-task model? Parallel heads avoid sequential propagation. However, Unified AES and GEC found that running intermediate reasoning first and scoring second beat parallel multi-tasking: grammar correction followed by scoring reached QWK 64.16, while the parallel setup did worse, though that result is within-prompt Korean data. The SICTS-style baseline tests whether this holds across prompts.
Why not just prompt an LLM for the score directly? TRATES and SICTS both report that zero-shot LLM scoring trails trained baselines. We include it only as a reference.
Why not just abstain on low-confidence traits, as Omethi abstains on low-confidence items? Hard dropping is one point on the range our gate covers, and we run it as an ablation: set $r=0$ below a threshold, otherwise 1. A continuous gate should help when trait reliability varies gradually across prompts.
Decisive Checks
Three results decide the claim. First, the share of the propagation gap explained by per-prompt offsets in the diagnosis stage. Second, whether prompt-held-out cross-fitting beats random-fold cross-fitting in holistic QWK on ASAP. Third, whether the label-free $r_{t,p}$ correlates with the true per-prompt trait QWK across ELLIPSE's 44 prompts.
If all three are positive and the full system exceeds 0.698 while also producing trait scores, the trait conundrum is resolved. If the first is negative, the hypothesis is wrong.
References
- Conundrums in Cross-Prompt Automated Essay Scoring: Making Sense of the State of the Art
- ICLE++: Modeling Fine-Grained Traits for Holistic Essay Scoring
- TRATES: Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring
- Relieving the Computational Bottleneck: Joint Inference for Event Extraction with High-Dimensional Features
- Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction
- Automated Cross-prompt Scoring of Essay Traits
- Improving Prompt Generalization for Cross-prompt Essay Trait Scoring from the Scoring-invariance Perspective
- Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring
- Graph-Based Multi-Trait Essay Scoring
- Down the Cascades of Omethi: Hierarchical Automatic Scoring in Large-Scale Assessments
- Argumentation: Content, Structure, and Relationship with Essay Quality
- Unified Automated Essay Scoring and Grammatical Error Correction