Dialect-Robust Domain Pre-Training for Rubric-Based Grading

Automated graders that check a student's answer one rubric point at a time are now accurate enough to take real work off instructors. On college chemistry answers, a fine-tuned RoBERTa model that treats each rubric item as a yes/no "does the answer imply this?" question reaches 86.8% accuracy. That question is natural language inference (NLI): deciding whether one text logically supports another. The same kind of NLI model, however, loses accuracy when the input is written in African American English rather than Standard American English. Negative inversion alone produces a 9.09% error rate on MNLI, a standard NLI benchmark. For grading, this failure has a direct cost. A student who explains the chemistry correctly in a non-standard dialect can be marked as missing a rubric point. The grader is then penalizing how the student writes, not what the student understands. No one has measured this effect in a grader. It also sits on a real methodological tension. The best-known recipe for domain-adapting a grading model throws out non-canonical student text before pre-training. Separate work on dialect robustness shows that exposing a model to non-canonical text is exactly what makes it robust.

This note proposes resolving that tension. The idea is to separate content quality from surface form during domain pre-training. Before fine-tuning a rubric grader, we continue pre-training it on domain text that is always factually correct but appears in several surface forms: the original Standard English, dialect-transformed versions, and character-noised versions. The bet is that the earlier "exclude bad student answers" result was about bad content, not unusual form. If so, a grader can keep its domain precision and stop converting dialect into lost points. The deliverables are a measurement of dialect-induced false negatives in rubric graders, which has never been reported, and a test of whether pre-training, rather than cheaper fixes, is where the repair belongs.

Core Claim

Domain continued pre-training on correct content in varied surface forms will shrink the gap between how often a grader credits a rubric point in a standard-English answer and how often it credits the same point in a meaning-equivalent dialect version. It will do this without lowering standard-English accuracy. It should also work better than adding dialect examples only at fine-tuning time, especially on dialect constructions the model never saw during training.

The claim rests on three pieces of prior work. Each plays a different role.

Pre-Training BERT on Domain Resources for Short Answer Grading supplies the method and also constrains it. That paper continues BERT's pre-training on textbooks and on pairs of questions with reference or correct student answers. It raises 3-way grading F1 from 74.55% to 76.25% (Government) and from 81.35% to 83.95% (Physiology). It also finds that incorrect student answers must be excluded from this stage. We keep that rule for content and drop it for form.

Different Perturbations, Different Mechanisms supplies the perturbation technique and a mechanistic diagnostic. It shows that continued pre-training with character-level noise improves zero-shot dialect transfer, in 43 of 45 settings, partly by pulling dialect and standard embeddings together. Its evaluations are limited to tasks like named-entity recognition and sentiment classification; it does not test evaluative judgments like grading.

VALUE supplies both the transformation tool and the motivation. Its 11 morphosyntactic rules convert Standard English into African American English. The paper also shows that fine-tuning on transformed data does not fully close the dialect gap on GLUE, a suite of general language-understanding benchmarks. That partial failure is exactly what our fine-tuning-augmentation baseline will re-test in the grading setting.

Research Program

We first build counterfactual grading test sets. These pair each original answer with dialect versions that mean the same thing. The sources are RiceChem, with 1,264 chemistry answers and 8,392 rubric-item labels, and the four-domain short-answer corpus. We apply VALUE's rules to every student answer. Ideally, speakers of African American English then check a sample to confirm two things: the transformation is natural, and every rubric label still holds. Because the rubric label should be unchanged by construction, any change in the grader's verdict is a dialect-induced error. This turns a hard fairness question into a direct measurement. Synthetic dialect is not authentic student writing, and the validated sample is our hedge. If funding allows, a small set of authentically written dialect answers to RiceChem prompts would be a stronger test.

Next we build the pre-training corpus. It holds college chemistry textbook text plus question-reference and question-correct-answer pairs, following the domain-BERT recipe. We create three surface variants of this corpus. The first is the original Standard English. The second applies a subset of VALUE rules. The third applies character noise in the style of Srivastava and Chiang. We continue pre-training RoBERTa-large-mnli on each mixture and then fine-tune it for rubric entailment exactly as RiceChem does. The VALUE rules are split into two groups. One group is used during training. The other group is held out and appears only at test time. This split separates genuine robustness from simply memorizing the transformations.

Finally, we test the mechanism and its consequences for selective human review. On the mechanism side, we measure the cosine similarity between representations of each answer and its dialect counterpart, to see whether gains come from aligning the two internal representations as the perturbation paper predicts. On the review side, SURE routes high-variance grades to humans. It saves 40–90% of grading labor, and its authors leave open whether some answer styles inflate uncertainty. We apply the same routing logic using disagreement across a five-seed ensemble of each grader. This tells us whether dialect answers are disproportionately sent to humans, or worse, confidently misgraded.

Baselines

The naive reference point is the RiceChem grader as published: RoBERTa-large-mnli fine-tuned for rubric entailment, with no domain pre-training and no dialect exposure. It is the strongest reported system on this task. The file reports 86.8% accuracy and 0.888 F1 for it (Automated Long Answer Grading with RiceChem Dataset). We first re-run it to confirm we reproduce that figure on the original test set. We then report its accuracy and its dialect flip rate on the counterfactual set. No reported result found on Lacuna for any grader on dialect-transformed grading data, so this flip rate is a new number we establish rather than one we can compare to.

The strongest domain-adaptation competitor is clean domain pre-training: the recipe from Pre-Training BERT on Domain Resources for Short Answer Grading applied to the same chemistry corpus with no surface variants. It runs with the same number of pre-training tokens and the same fine-tuning schedule as our model. On the four-domain corpus, the reported reference points are 76.25% F1 (Government) and 83.95% F1 (Physiology).

The strongest robustness competitor is fine-tuning-time augmentation. Here the RiceChem grader is fine-tuned on its original training answers plus their VALUE-transformed copies, with the same rules available as in our pre-training. All systems receive equal tuning budgets of 20 hyperparameter trials. Systems with pre-training stages are additionally matched in total compute.

Two ablations isolate the key ingredient. One applies perturbations during pre-training but uses only general web text, which tests whether domain content matters. The other uses domain text but only character noise, which tests whether dialect-specific rules matter.

To count as beating the competitors, our model must hit three targets. It must be within one point of clean domain pre-training on original-test accuracy. It must reduce the flip rate on the held-out dialect rules relative to both fine-tuning augmentation and clean domain pre-training. And this reduction must hold across five seeds, with a non-overlapping bootstrap confidence interval.

Why not just…?

Why not just augment the fine-tuning data with dialect-transformed answers? This is the real threat, because it is cheap and directly targets the failure. VALUE reports that augmentation of this kind narrowed but did not close the gap on GLUE. That evidence comes from general language understanding, not grading, so it is suggestive rather than decisive. The held-out-rule split settles the question. Fine-tuning augmentation will probably fix the constructions it was trained on. The open question is whether it generalizes to unseen ones as well as pre-training-time exposure does. If it matches our method on held-out rules, we will report that the cheaper fix suffices and that the pre-training story is unnecessary.

Why not just translate student answers into Standard English before grading? Linguistic Bias in ChatGPT shows that models flatten dialect features toward standard varieties. Jamaican English features survive only 2% of the time. That flattening is exactly the risk in a normalization step: it can quietly drop or rewrite the content that a rubric point depends on. Equity Beyond Bias in Language Technologies for Education adds that this kind of enforced assimilation is itself a representational harm. To test the approach rather than dismiss it, we include a normalization baseline. An open instruction-tuned model rewrites each dialect answer into Standard English, and the RiceChem grader then scores the rewrite. We measure its flip rate. We also have annotators check a sample of rewrites to see whether rubric-relevant content was lost in rewriting.

Why not just prompt a frontier LLM with the rubric and route uncertain cases to humans? On RiceChem, zero-shot GPT-4 reached only 70.9% accuracy, well below the fine-tuned encoder. Single-prompt LLM grading also systematically underscores students (SURE). The same dialect study cited above found GPT-4 18% more likely than GPT-3.5 to produce stereotyped content when imitating minoritized varieties. That finding is from a different task, so here we apply the selective-review analysis to our encoder graders and ask directly whether review routing is equitable.

Proposed Experiments and Decisive Checks

The primary metric is the dialect flip rate. It is the fraction of rubric items credited in the original answer but not credited in its dialect counterpart, computed separately for training-rule and held-out-rule transformations and for the validated subset. We also report the difference in false-negative rate between original and dialect answers. Secondary metrics are standard-English accuracy and F1 on RiceChem and macro-F1 on the four-domain corpus.

The decisive result has three parts. Our model must lower the held-out-rule flip rate below that of fine-tuning augmentation. It must do so at no more than a one-point accuracy cost relative to clean domain pre-training. And the gain must track increased similarity between original and dialect representations, which would point to the representation-alignment mechanism.

There are two possible falsifications. If perturbation during domain pre-training hurts standard-English accuracy, the earlier "exclude non-canonical text" rule is about form after all, and the trade-off is real. If fine-tuning augmentation matches our model on held-out rules, pre-training is the wrong place for the fix. The seed-ensemble routing analysis then tells practitioners something either way: whether review systems like SURE send students writing in African American English to human graders at a different rate, and whether domain pre-training changes that rate.