Judging Grammar Edits in Partnerships, Not One at a Time

When a learner writes "The apple are red," a correction system might change "apple" to "apples," or "are" to "is." Either fix alone is fine. But a system that proposes "apples are" has made two edits that only make sense together, and a system that proposes "apples is" has made two edits that each look reasonable but together produce a new error. Grammatical error correction, the task of rewriting learner text into fluent, correct text, is full of these linked errors, sometimes called "partners in crime." Yet the newest automatic evaluators judge each edit by itself. They use large language models as graders, an approach called LLM-as-a-judge. Because they look at one edit at a time, they can call a correct verb fix wrong because its partner noun fix is not applied in the context they see, or call an edit right when it breaks agreement with a neighbour.

We propose to evaluate grammar edits as interacting groups. The judge measures whether an edit's quality depends on the presence of other edits, groups edits whose effects do not add up independently, and then rules on validity and credit at the level of the group. Two supporting pieces make this workable. The first is a retrieval step that shows the judge past examples of the same kind of linked error. It combines exact matching on error-type signatures with meaning-based matching, a combination called hybrid retrieval. The second is a statistical correction that turns noisy group-level verdicts into trustworthy system-level estimates. In short, the paper argues that edit-level evaluation is wrong on the edits that interact, shows how to detect those edits cheaply, and shows that fixing this changes both metric reliability and the quality of training data built by automatic reference expansion.

Core Claim

Edit-level validity is not additive for interdependent edits, and current per-edit judges misjudge them in a systematic, measurable way. Two recent lines of work score edits one by one. JELV runs a multi-turn ensemble of LLM judges that rule on each edit's grammaticality, faithfulness and fluency, reaching over 90% agreement with experts on PEVData. It then uses those verdicts to expand the set of reference corrections used for training. Goto et al.'s edit-level attribution treats edits as players in a cooperative game and splits a sentence-level score among them with sampled Shapley values. It reports roughly 70% alignment with human edit judgments, and it names co-dependent edits such as subject–verb shifts as its main unsolved limitation. Both papers motivate this proposal: they expose the gap without closing it.

We close it with a counterfactual interaction test. Let $V(S)$ be the judge's quality score for the source sentence with only the edits in subset $S$ applied. For two edits $e_i$ and $e_j$, the pairwise interaction is

$$I(e_i, e_j) = V(\{i,j\}) - V(\{i\}) - V(\{j\}) + V(\emptyset).$$

This is the pairwise Shapley interaction index. It is zero when the two edits contribute independently, positive when they only help together, and negative when they conflict. Edits with large $|I|$ are merged into clusters, and validity and credit are then assigned per cluster. The method treats the joint-inference view of Rozovskaya and Roth as a judging problem rather than a decoding problem. That work enforced global consistency between correction modules with integer linear programming and raised NUCLE F1 from 31.20 to 35.21, and subject–verb agreement F1 from 39.64 to 52.25. Its gains depended on a filter that restricted joint reasoning to words within 1–3 tokens, because parsers break on learner text. Our interaction test needs no parser, so it can find partners at any distance.

Research Program

The first component is a diagnostic dataset of partner edits. We take sentences from CoNLL-2014 and NUCLE whose references contain two or more edits. We enumerate edit subsets up to a cap, typically 2–5 edits per sentence, and generate the partially corrected sentences. Annotators then label three things: each edit's validity in isolation, each edit's validity with its candidates for partners applied, and which edits belong together. We also inject "broken partnerships" by applying only one edit of a known agreement cluster. This lets us measure exactly how often per-edit judges approve half a fix. We target about 1,500 sentences, which is small enough for careful annotation. It is our main human-labeled resource.

The second component is the cluster judge itself. Computing $I$ for every pair costs a number of judge calls that grows with the square of the number of edits. That is affordable at typical edit counts, but we add a cheap first pass. A distilled small judge in the style of JELV2.0, a DeBERTa model with perplexity and sentence-embedding features, screens pairs. Only pairs flagged as likely interacting go to the expensive multi-turn judge. This follows the budget concerns raised in LLM-as-a-Judge on a Budget.

The third component is hybrid retrieval of demonstrations for the judge. Here retrieval plays a supporting role rather than serving as the core idea. Explanation-based demonstration retrieval for GEC showed that meaning-based (dense) retrieval picks examples that share a topic rather than an error type. It also showed that explanations help most when used as retrieval queries, not when pasted into the prompt, an effect the authors call the "CoT Paradox." That work used dense matching only, over one flat explanation string per sentence. We index past judged clusters by a sparse error signature, meaning exact tokens for the ERRANT error-type combination and the involved lemmas (for example, noun-number plus subject–verb agreement on "apple/be"). We combine that with a dense embedding of the context, scoring candidates by $\alpha\,s_{\text{sparse}} + \beta\,s_{\text{dense}}$. This is the linear interpolation that Luan et al. showed preserves exact-match fidelity while adding semantic reach. Following the CoT Paradox finding, the retrieved clusters are shown to the judge as verdicts only, without their explanations. Retrieval is the ingredient most likely to turn out unnecessary, so it is ablated explicitly.

The fourth component is calibrated aggregation. Cluster verdicts from an LLM judge carry systematic, asymmetric errors. To compare systems by their rate of valid clusters, we combine judge labels with the human-labeled subset using the PPI++ estimator studied in Efficient Inference for Noisy LLM-as-a-Judge Evaluation. That paper reported confidence intervals 35–55% narrower than standard prediction-powered inference on Chatbot Arena preferences, and it lists structured, instance-dependent judging as an open extension. Clusters are such a structured setting.

Baselines

The naive reference point is ERRANT-style reference matching. It needs no judge, and it is the baseline that motivated reference expansion in the first place. On the SEEDA-S Base set, ERRANT's Pearson correlation with human system rankings is 0.545 under standard aggregation and 0.763 under TrueSkill aggregation (Goto et al., Rethinking Evaluation Metrics). We report it under both aggregation schemes so that differences in aggregation are not mistaken for differences in judging.

The strongest existing edit-level competitor is JELV. It is the only system in the file that uses LLM judges for edit validity and also turns those judgments into expanded training references. It reports over 90% human agreement on PEVData and a Pearson correlation of 0.975 for edit-level evaluation on SEEDA's social-media portion. For fairness we run the same three base models through the same explain-then-annotate protocol. The only difference is that we feed it single edits rather than clusters, which makes the comparison isolate the unit of judgment. We also give it the same number of judge calls per sentence by letting it re-judge each edit with extra self-consistency samples. To beat JELV we must show significantly higher agreement with human labels on the partner-edit subset of our diagnostic set while staying within one point on non-interacting edits. If we cannot match it on ordinary edits, the method is not a usable replacement.

The strongest sentence-level competitor is IMPARA with TrueSkill aggregation, at 0.975 Pearson on SEEDA-S (+Fluency). We do not expect to beat it on system ranking. The claim is that we match its ranking quality while also providing correct edit-level verdicts, which IMPARA does not attempt.

For attribution, the baseline is Goto et al.'s Shapley sampling with roughly 70% human alignment. We give it the same judge as its value function, so that any gain comes from clustering rather than from a better scorer.

For downstream correction, we retrain GECToR and SynGEC on references expanded by JELV and by our cluster judge, then evaluate on CoNLL-2014. JELV reports only "consistent performance improvements" over raw data, so no reported number was found on Lacuna for the size of its CoNLL-14 gain, and we will reproduce it ourselves. As an external anchor, explanation-based retrieval reports a CoNLL-14 F0.5 of 59.38 with DeepSeek2.5.

Three ablations isolate the key ingredients. The first replaces the interaction test with Rozovskaya and Roth's 1–3 token proximity rule for forming clusters. The second removes retrieval, and also compares sparse-only and dense-only retrieval against the hybrid. The third replaces PPI++ with plain averaging of judge labels.

Why not just…?

Why not just give the judge all the edits at once and ask it which ones belong together? This is the most serious threat. A strong model might group edits well from a single prompt, at a fraction of the cost. We run this single-prompt joint judge as a main comparison, with the same models and the same budget. Our reason to expect a shortfall is the evidence that flat, holistic judging misses propagated errors. AgentEval found that structured, dependency-aware step judging raised failure-detection recall from 0.41 to 0.89 on agent traces, where 63% of failures propagated from upstream steps. That is a different domain, though. If the single-prompt judge matches the counterfactual test on the partner-edit set, we will report that, and recast the contribution as the diagnostic dataset plus cluster-level attribution and calibration, with the cheap joint prompt as the recommended clustering method.

Why not just group edits that are within a few words of each other? Proximity is what made joint inference precise in Rozovskaya and Roth, and most agreement errors are local. That paper also attributes its limits to dependencies beyond three tokens, where parsers fail on ungrammatical text. Our diagnostic set records the token distance of every annotated partnership. The proximity ablation will show how much of the gain comes from long-range clusters. If almost all real partnerships are local, the interaction test only matters for a small group of edits, and we will say so.

Why not just score whole sentences and skip edit-level judging? Sentence-level metrics with TrueSkill aggregation already rank systems nearly as humans do (0.975 for IMPARA). But reference expansion and learner feedback need to know which edits are valid. A sentence score cannot say whether "apples is" contains one good edit or zero. The reference-expansion experiment tests whether that edit-level correctness pays off in training data.

Why not just use dense retrieval, or no retrieval at all, for the judge's examples? Dense retrieval tends to match topic rather than error type (explanation-based retrieval). The sparse error signature is designed to fix exactly that. The retrieval ablation settles whether examples matter at all. If removing retrieval costs nothing, we drop it from the method.

Decisive Checks

The first decisive result is the broken-partnership rate: how often JELV-style per-edit judges approve one half of an agreement cluster when the other half is missing, compared with the cluster judge. If per-edit judges rarely make this mistake, the premise fails and the project should stop there.

The second is agreement with human labels, broken down by interaction strength. Gains should concentrate where $|I|$ is large and vanish where it is near zero. That pattern is the signature that the mechanism, rather than extra compute, is responsible for the improvement.

The third is whether CoNLL-2014 F0.5 for GECToR and SynGEC improves when they are trained on cluster-filtered expanded references rather than JELV-filtered ones, with the same expansion budget.

The fourth checks that PPI++ intervals on cluster-validity rates keep their nominal coverage on held-out human labels while being narrower than those from human labels alone.

All four are run on CoNLL-2014, NUCLE, SEEDA and PEVData, plus the new partner-edit set.