Which Essays Should a Scorer Compare To? Trait-Specific Exemplar Retrieval for Error-Laden Student Writing

One way to have a language model grade an essay is to show it a few already-graded essays and ask it to score the new one by analogy. This is called in-context learning: the model conditions on example pairs in its prompt and receives no weight updates. How well it works depends on which examples are shown, so a common practice is to retrieve the stored examples most similar to the new input. Student essays are a hard case for that retrieval step. They are full of misspellings, and misspellings break how models read text. Models split words into vocabulary pieces called subwords, and a single wrong letter turns one familiar piece into several rare fragments. The essay's embedding then drifts, and the retriever returns the wrong neighbors. Rubrics add a second complication, because they grade essays on separate traits such as Content, Organization and Conventions. Spelling errors are noise when judging Content, but they are the evidence when judging Conventions. No single notion of "similar essay" can serve both traits.

We propose that retrieval for multi-trait scoring should be trait-specific. For Content and Organization, exemplars should be retrieved in a view that ignores spelling. For Conventions and Language, they should be retrieved in a view that matches the essay's error profile. Part of that profile is measured directly from how badly the tokenizer fragments the writing. The claim is testable because it predicts a crossover. A spelling-invariant retriever should help Content and hurt Conventions relative to a raw retriever, and an error-profile retriever should do the reverse. If one retriever wins on every trait, the idea is wrong.

Core Claim and Positioning

Retrieval-based prompting has mostly been studied where similarity is a single, task-level notion. LM-BFF picks demonstrations nearest to the input under Sentence-BERT and reports an 11% average gain over fine-tuning in few-shot classification. Dr.ICL trains a retriever with a language model's likelihood of the correct answer and gains 4.7 points one-shot on GSM8K. Both methods supply our machinery, but both assume one similarity per task. We replace that assumption with one similarity per trait.

Essay-scoring work that uses large language models does not retrieve external graded exemplars at all. RMTS and TRATES use fixed prompt templates to generate rationales or rubric sub-questions. MOOSE uses the essay as a query, but only to attend over the prompt and the essay's own chunks. These papers establish the application setting and leave external, trait-conditioned exemplar retrieval open.

The fragility literature motivates the problem. Heigold et al. show word scrambling collapsing a subword-based translator from 31.6 to 3.6 BLEU. Char2Subword shows that composing subword embeddings from characters shrinks adversarial accuracy loss from 16.78 to 12.41 points. Neither paper measures what shattering does to retrieval neighborhoods.

The closest robustness study, On the Noise Robustness of In-Context Learning for Text Generation, corrupts demonstration labels rather than query spelling. It constrains our design: it finds that adding more demonstrations amplifies damage when those demonstrations are bad. Poor neighbors will therefore not be averaged away by raising $k$.

Finally, Singla et al. show that holistic scorers keep 95% of their agreement after large deletions yet swing up to 50% from 3–5 trigger words. That study only examined single-score models. We ask the trait-level version of the question.

Research Program

The system has three parts.

Graded exemplar index. We build an index of essays with per-trait human scores from prompts other than the test prompt, so that evaluation is cross-prompt.

Two retrieval views.

The invariant view is a dense encoder that should place a misspelled essay next to its clean counterpart. It is trained in two steps:

  • It is first continually pre-trained on unlabeled learner writing (ASAP, ELLIPSE and DREsS essay text). This follows the "drowning effect" argument of Durrani et al. that specialized text distributions need isolated training.
  • During that pre-training, tokenizations are randomly split in the style of StochasTok, which retrofitted subword robustness into GPT-2 in 2,000–7,000 steps without benchmark loss.
  • It is then contrastively tuned so that each essay and a character-perturbed copy map to the same point.

The error-profile view represents an essay by its surface error statistics. These include:

  • tokens per word relative to a clean reference;
  • the fraction of words that shatter into three or more pieces;
  • the distribution of edit types from a grammatical-error-correction tagger.

Neighbors under this view are essays with similar kinds and densities of errors, regardless of topic.

Trait-specific retrieval and scoring. Each trait is assigned a view, and a Dr.ICL-style tuning step optionally adapts that view's retriever. The tuning signal is the scoring model's likelihood of the gold score for that trait. A fixed open model (Llama-3.1-8B, used as a teacher in RMTS) then scores each trait in its own prompt with $k$ retrieved exemplars.

The analysis component comes first and stands on its own. We measure how much retrieval neighborhoods move under controlled character noise and under naturally occurring misspellings. Stability is neighbor overlap at $k$ between clean and noisy queries. Downstream harm is the per-trait loss in agreement.

Baselines

Floor baselines. The naive reference is zero-shot trait scoring by the same Llama model with the rubric only. The next step up is static prompting: a fixed set of $k$ graded essays, sampled stratified by score, reused for every query.

Trait-agnostic retrieval. This is the direct competitor to our key ingredient. It retrieves $k$ exemplars by off-the-shelf Sentence-BERT similarity on the raw essay, in the manner of LM-BFF, and uses the same exemplars for every trait. A trait-agnostic Dr.ICL-style trained retriever is also included, since that is the strongest existing way to choose demonstrations.

Isolating ablations. Four ablations separate the ingredients:

  • a single retriever built on the invariant view and used for all traits;
  • a single retriever built on the error-profile view and used for all traits;
  • trait-specific retrieval using the off-the-shelf encoder, so without learner-text pre-training;
  • trait-specific retrieval with the learner-text pre-training but without stochastic segmentation.

Fair-comparison conditions. All retrieval arms share:

  • the same index;
  • the same scoring model;
  • the same $k$ and prompt format;
  • the same number of retriever-tuning steps.

Strongest trained scorers. The strongest overall competitors are purpose-trained cross-prompt trait scorers, reproduced with their released settings on the same leave-one-prompt-out folds. ProTACT reports 0.592 average cross-prompt Quadratic Weighted Kappa (QWK, an agreement score that penalizes large disagreements more) on ASAP/ASAP++. CTS reports 0.553. RMTS reports 0.648 QWK for Pegasus with LLM rationales on ASAP/ASAP++. The file does not make clear whether that RMTS number is cross-prompt, so we rerun it on our folds rather than quote it as comparable. For retrieval-based in-context essay scoring itself, no reported result was found on Lacuna for ASAP, ELLIPSE or DREsS.

What counts as beating each baseline. Beating trait-agnostic retrieval means a significant per-trait QWK gain, measured with prompt-level bootstrap. We also require the predicted crossover across traits. Matching ProTACT's 0.592 without training a scorer would be a practically meaningful result. Exceeding it is not required for the core claim.

Why not just…?

Why not just spell-correct the essay before retrieval? This is the real threat, and we treat it as an experimental arm rather than a strawman. Running a correction model on the query and retrieving on the corrected text may give an invariant view as good as our trained encoder. If it does, we adopt it, and the contribution reduces to the trait-specific split, which we still expect to matter. Correction applied uniformly erases exactly the evidence the Conventions trait needs. The decisive comparison is therefore three-way on Conventions:

  • corrected-text retrieval for all traits;
  • corrected text for Content plus raw error-profile retrieval for Conventions;
  • our full system.

Why not just train a multi-trait scorer and skip retrieval? Trained scorers like ProTACT and T-MES are strong. However, they inherit the oversensitivity Singla et al. document, and they give a teacher no comparable essays to justify a grade. We run the trigger-word attack from that paper against both families. If trained scorers win on clean QWK but retrieval-grounded scoring moves less under triggers and misspellings, that is a meaningful trade-off, not a loss.

Why not just use a character-level or Char2Subword encoder as the one retriever? A spelling-robust encoder addresses drift but still imposes one similarity on all traits. The "invariant view for all traits" ablation is exactly this system. The prediction is that it improves Content and degrades Conventions. If it instead improves both, trait-specific retrieval is unnecessary and we will say so.

Proposed Experiments

Datasets. The main benchmark is ASAP/ASAP++ with leave-one-prompt-out folds over the eight prompts and five shared traits. ELLIPSE, with 44 prompts, tests transfer across many topics. DREsS_New (2,279 expert-graded EFL essays) tests whether the Language trait behaves like Conventions.

Controlled noise. Test essays receive character insertions, transpositions and substitutions at 0–20% of words. Infix corruption and boundary corruption are separated, following the finding cited in the file that infix noise is more damaging. Clean and noisy runs share gold scores. For natural noise, essays are stratified by measured fragmentation rate.

Metrics. We report per-trait QWK, neighbor overlap at $k$, and the score shift under Singla-style triggers.

First decisive check. This is the trait-by-view interaction: a two-way analysis of QWK over trait and retrieval view, with prompt as the resampling unit. The hypothesis requires opposite-signed effects for Content and Conventions.

Second decisive check. This is dose-response. As injected noise rises, neighbor overlap for trait-agnostic Sentence-BERT retrieval should fall. The Content-QWK gap between it and the invariant view should widen in step. If overlap stays high under noise, then shattering does not disturb retrieval enough to matter, and the motivation fails.

Third decisive check. Because of the demonstration-noise result, we sweep $k$ from 1 to 8. The prediction is that trait-agnostic retrieval degrades with larger $k$ on noisy queries, while trait-specific retrieval does not.