Large language models are increasingly used as automatic graders for grammar and spelling correction. The judge reads a sentence and a proposed edit and decides whether the edit is valid. Practitioners trust these judges when they agree with humans. When the judge wavers, the usual fix is to sample it several times and take a vote. This note argues that both habits rest on a hidden assumption: that the text looks like the benchmarks the judges were checked on. Nearly all of those benchmarks come from second-language learners' essays. Native writers make different mistakes. Their errors are sparse and fluent-looking: a homophone picked from a Chinese pinyin keyboard, a Korean word-spacing slip, a missing Arabic punctuation mark. Learners make dense structural mistakes. We hypothesize that judges fail on native errors in a particular way. They are not noisy on this text. They are stably wrong, because a fluent sentence with one slip looks fine to a model that rewards fluency. If that holds, vote-based reliability checks will stay silent exactly where the judge is worst.
The proposal has two parts. First, we build a paired native-versus-learner audit of LLM judges from existing correction corpora in several languages, and measure accuracy, run-to-run instability, and calibration separately for each writer population. Second, if the hypothesis holds, we test a label-free probe for languages that have no native-error data. The probe injects synthetic native-style slips into clean text and measures how often the judge misses them. The result would be a per-population, per-language reliability estimate, where current practice reports a single agreement number from learner data.
Core Claim
Consider an LLM judge that checks the edits proposed by a text-correction system. Its failures on native-speaker text are dominated by consistent errors: the same wrong verdict across repeated samples and candidate orderings. Its failures on learner text are dominated by unstable ones. Two consequences follow. Instability-based signals such as vote fraction, flip rate under reordering, and self-consistency will flag learner-text judge errors well but native-text judge errors poorly. And judge agreement measured on learner benchmarks will overstate reliability on native writing, with a gap that varies by language and by the kind of error each language's native writers produce.
This claim connects the four directions. The difference between learner and native errors is the mechanism. LLM-as-a-judge is the object being studied. Output variability is the diagnostic we test and expect to fail. The multilingual evaluation gap is the setting that makes a label-free fix necessary.
Why This Is Open
The motivating evidence comes from the correction literature. NaSGEC shows that Chinese correction systems trained on learner data collapse on native text. It also reports that native sentences are much longer than learner sentences: 54.3 versus 29.7 tokens against the learner benchmark NLPCC18. Flachs et al. show that correction models over-correct when errors are sparse, which is the typical condition in native writing. CSCD-NS documents that native Chinese spelling errors are driven by sound similarity through keyboard input rather than by visual similarity. ZAEBUC finds learner Arabic word error rates about 50% higher than native ones, with qualitatively different error types. The Korean standardization work of Yoon et al. likewise separates native spacing errors from learner errors in particles and verb endings. All of this work studies correction systems. None of it asks whether the judges now used to grade those systems inherit the same blind spot.
The closest work on the judge side supplies our method but not our question. JELV uses an ensemble of frontier LLMs to judge whether a correction edit is grammatical, faithful and fluent. It does not stratify its reliability by writer population, and the file we have does not tie it to a native benchmark. How Reliable is Multilingual LLM-as-a-Judge? and Dialectal Toxicity Detection show that judge consistency drops outside standard English. They measure consistency, whereas our hypothesis concerns errors that consistency cannot see. LLMBar shows judges favoring pleasant surface form over correct content. This "pretty trap" is the likely cause of stable errors on fluent native text.
On the variability side, Renze finds a stability plateau in multiple-choice accuracy up to temperature 1.0. Ouyang et al. show that even temperature-0 output drifts, and that typo perturbations drop sentiment accuracy from 80.6% to 61.3%. Both treat variance as the problem to be reduced. We treat low variance as a possible false comfort.
Research Program
The audit set. We build an edit-level judge audit from corpora that contain both native and learner text. For Chinese, we pair native sources (NaSGEC, CSCD-NS, and CCTC for cross-sentence cases) with the learner benchmark NLPCC18. For Korean we use the native and learner portions of Yoon et al.; for Arabic, the native and learner writers in ZAEBUC; for English, learner essays from the standard benchmarks plus native text.
Norwegian is a partial case. NoCoLA draws on learner essays from the ASK corpus, so a matched native Norwegian source would have to be found. If none is found, Norwegian serves only as a learner-side reference.
Each item is one of two kinds. The first is a source sentence plus a candidate edit produced by a correction system, with sequence-to-sequence and edit-based systems both used, since they fail differently. The second is an uncorrected sentence the judge must accept or flag. Gold references are known to be incomplete, which is exactly JELV's motivation. So a stratified subset of roughly one thousand items per language-and-population cell is adjudicated by trained native-speaker annotators. That subset is the ground truth. Items are balanced on sentence length and edit count so that length effects, which NaSGEC documents, do not masquerade as population effects.
Measuring variability properly. Each judge sees every item ten times at temperature 0 and ten times at 0.7, with both candidate orderings wherever a comparison is involved. We use open-weight judges with fixed seeds and tensor parallelism disabled, alongside API judges, so that sampling randomness can be separated from serving non-determinism.
For each item we compute $u(x) = 1 - v(x)$, where $v(x)$ is the fraction of runs giving the majority verdict. The key quantity is the stable-wrong share: among items the majority vote gets wrong, the fraction with $u(x) \le 0.1$. We also compute the area under the ROC curve for $u(x)$ as a detector of judge error, and the expected calibration error of the vote fraction and of the judge's token probability. Every number is reported per cell.
A label-free probe for uncovered languages. If the stable-wrong effect appears, we need to estimate it where no native annotations exist. We borrow the logic of Clouâtre et al., who map where a model fails to understand a language by perturbing unlabeled text and watching the response. Here we inject native-style slips into clean monolingual text, taken from MADLAD-400, and measure how often the judge accepts the corrupted sentence. The slips are homophone substitutions, spacing and punctuation deletions, and agreement flips generated with MultiBLiMP-style morphological templates. The pseudo-native generators of NaSGEC and Ma et al. supply the injection recipes. NaSGEC also supplies a warning: synthetic errors fail to reproduce deliberately crafted exam errors. The probe therefore estimates a lower bound on miss rates, and we validate it only in languages where the audit set gives real numbers.
Baselines
The naive reference point is single-call judging at temperature 0, with judge reliability taken from agreement on learner data and applied unchanged to native text. This is what most papers do. The audit tests it directly: if per-population accuracy on native text is not meaningfully lower, the premise fails.
The strongest existing method is the JELV-style frontier ensemble from JELV, which combines several frontier judges with explain-then-annotate prompting over grammaticality, faithfulness and fluency. It is the strongest competitor because it already uses both disagreement across models and reasoning, which are the two things most likely to rescue native slips. We run it with its published prompts on the same audit items. It gets the same number of total model calls as our repeated-sampling runs, and the prompt budget is matched across conditions. No reported result was found on Lacuna for JELV accuracy on native versus learner edits, so our own run provides the reference. The judge-side figure the file does give is GPT-4 at about 73% accuracy on LLMBar's adversarial pairs (LLMBar). It is a different task and serves only as context.
Self-consistency voting, following Ruan et al. and Kim et al., is the variance-based baseline whose failure the core claim predicts. We also include a distributional judge in the spirit of Chen et al., which predicts a spread over verdicts rather than one answer, run at matched calls where training data permits.
The ablation that isolates the key ingredient keeps judge, prompts and budget fixed and changes only the population of the text. Any gap therefore belongs to writer population, not to method.
Beating the baselines has a precise meaning. The first claim holds if the stable-wrong share on native items is significantly higher than on learner items for every strong judge, including the ensemble, and instability AUROC is significantly lower on native items. The probe succeeds if its predicted miss rates rank languages and error types in the same order as the audit, with a rank correlation clearly above zero.
Why not just…?
Why not just take a majority vote over many samples? Voting helps only if errors are unstable. The core claim is that native-text errors are not. Voting over ten runs of a stable-wrong verdict returns the wrong verdict with high confidence. The audit tests this directly: if the stable-wrong share is similar across populations, voting is adequate and our claim is refuted.
Why not just use the judge's token probability as confidence? This is the most serious threat. It is far cheaper than sampling, and on its own it could make our reliability probe unnecessary. We report its calibration error separately by population. If token probability stays well calibrated on native text, the right conclusion is that cheap confidence suffices and only the stratified audit remains a contribution. Our expectation, grounded in the fluency preference documented by LLMBar, is that probability will be high on fluent native sentences whether or not they contain a slip.
Why not just use a bigger judge or an ensemble? Scale and model diversity reduce independent mistakes. They do not fix a mistake that all the models share, and a preference for fluent text is plausibly shared. The JELV-style ensemble is in the baseline set for this reason. If it closes the native gap, we report that judges should be ensembles for native text, and the probe then becomes a cheap check of when that is needed.
Why not just translate an English learner meta-evaluation set? Translation keeps learner error types, which are then cast into another language, and adds errors from the translation system itself (XQA). Judges also show a preference for translated text (translationese bias). Translation would reproduce exactly the population mismatch we are measuring.
Why not just fine-tune a judge on native error data? In most languages that data does not exist, which is the multilingual gap itself. The native data that does exist is often pre-filtered by automatic tools, which removes the errors those tools miss (NaSGEC). We treat fine-tuning as a follow-up for Chinese, where the data allows it, rather than as a general answer.
Decisive Checks
The first check is the population contrast under matched length and edit density. Does the stable-wrong share rise, and instability AUROC fall, on native items in Chinese, Korean and Arabic for every judge tested? If the effect appears in only one language, it is a property of that language's errors rather than of native writing in general. We would report it that way.
The second check separates sampling randomness from serving randomness. Comparing seeded open-weight runs with API runs tells us whether the "stability" we observe is real stability of the judge or merely a quiet serving path, the kind of drift Ouyang et al. warn about. Judge accuracy on the two kinds of text, native and learner, must also be compared at both temperatures, since Renze's plateau was established on multiple choice, not on judgment tasks.
The third check validates the probe. Its predicted miss rates must track audited miss rates across languages and error types. It must also be shown not to transfer to cross-sentence native errors such as those in CCTC, which synthetic injection cannot mimic. That tells users where the cheap estimate stops being trustworthy.
A negative result on the first check is itself useful. It would license the field's current practice of reusing learner-derived judge reliability on native text.
References
- NaSGEC: a Multi-Domain Chinese Grammatical Error Correction Dataset from Native Speaker Texts
- Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses
- CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers
- ZAEBUC: An Annotated Arabic-English Bilingual Writer Corpus
- Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation
- JELV: A Judge of Edit-Level Validity for Evaluation and Automated Reference Expansion in Grammatical Error Correction
- How Reliable is Multilingual LLM-as-a-Judge?
- Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties
- Evaluating Large Language Models at Evaluating Instruction Following
- The Effect of Sampling Temperature on Problem Solving in Large Language Models
- Stability Analysis of ChatGPT-Based Sentiment Analysis in AI Quality Assurance
- CCTC: A Cross-Sentence Chinese Text Correction Dataset for Native Speakers
- NoCoLA: The Norwegian Corpus of Linguistic Acceptability
- Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
- MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
- MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
- Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correction
- PAI at SemEval-2025 Task 11: A Large Language Model Ensemble Strategy for Text-Based Emotion Detection
- Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoning
- Distributional LLM-as-a-Judge
- XQA: A Cross-lingual Open-domain Question Answering Dataset
- Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck