Peer review is science's first correction mechanism: a reviewer names a flaw, the authors revise, and someone has to check whether the flaw was actually fixed. Automating that check is hard for a simple reason. The only large public records of revision come from journals that publish transparent reviews of accepted papers. In that data almost every concern ends up "resolved," and the author's response letter almost always says so. A model trained on it learns to believe the response letter. We propose to manufacture the missing failures from the real data itself. We take a concern the authors genuinely fixed, remove their fix by restoring the original manuscript text, keep the response letter that claims the fix, and label the result "unresolved." Every word in the example is real and written by the authors. What changes is whether the manuscript contains the change. The label follows from how the example was built, not from a generator's guess.
The core claim is that this kind of data augmentation, meaning training examples built by recombining real before-and-after manuscript text, targets the specific failure that stops current revision verifiers. Generated synthetic failures do not target it. The augmentation forces a model to look in the manuscript for evidence, rather than rewarding it for reading the letter fluently. If the claim holds, a 9-billion-parameter open model trained this way should beat frontier models on the verification step of an existing benchmark. It should also transfer to real, naturally occurring failed revisions that it never saw as training labels.
The gap this fills
The problem is set up by AutoSupervision. That work builds a benchmark of 8,790 reviewer concerns across 847 revision episodes from Nature Communications. It splits the task into three parts: characterizing the concern, deciding whether it was resolved, and grounding the decision in specific manuscript text. It finds that frontier models characterize concerns well but verify them poorly. GPT-5.5 scores 0.754 on characterization and 0.501 on verification. The authors trace much of the gap to what they call the response letter trap, where authors overstate their fixes. They also note that the data barely contains failed revisions, because only accepted papers are published. FirstPass, which predicts revision-cycle length from 3,668 Nature Communications review dialogues, shares the same acceptance bias. Both papers motivate this proposal: they locate the bottleneck and its data cause, but neither attempts to fix the data.
The method draws on a lesson from augmentation research that is easy to misapply to text. Cut to the Mix shows that simple patch recombination (CutMix) beats elaborate, realistic augmentation for organ segmentation, even when the mixed images look anatomically wrong. The survey of mix-based augmentation frames such mixing as a regularizer that smooths decision boundaries. Two papers constrain how far that lesson carries into scientific text. Fact or Fiction shows that claim verifiers break on direction words such as "impaired" versus "increased," so edits that alter meaning silently corrupt labels. Articulate Intuition or Genuine Analysis? shows that LLM judges reward analytical form: length explains about 74% of the gap between AI and human review scores. Taken together, these results suggest that recombination is worth keeping and that generating new text is risky. We therefore recombine only real text blocks, at the level of whole manuscript passages. Labels come from the structure of the edit, not from meaning-altering rewrites.
Core Claim
Real revision episodes contain a before-version, an after-version, a response letter, and concern-to-edit links. Three counterfactual operations on these episodes yield label-exact negative and partial examples. Reversion restores every revised block tied to a concern to its original text, while keeping the response letter. This produces "unresolved" examples in which the letter lies. Misattribution leaves a real, substantive revision in place but pairs the concern with the edit made for a different concern in the same paper. The manuscript visibly changed, but not in response to this concern. This defeats the shortcut "text near the concern changed, so it is resolved." Partial reversion restores a subset of the blocks tied to a multi-part concern, producing "partially resolved" examples. Grounding labels come for free: the correct evidence span is the reverted block, or its absence.
Our hypothesis is that reversion alone teaches the model to detect whether text changed at all, and misattribution is what teaches it to check whether the change addresses the concern. The ablations are designed to test exactly this split.
Research Program
The first stage is episode alignment. We align original and revised manuscripts at the paragraph level, using the kind of paragraph-to-review linking that NLPeer builds as intertextual graphs, and link each reviewer concern to the edited blocks that the response letter cites. Only concerns with confident alignments are used to build counterfactuals.
Label noise is the main risk. A fix may live in supplementary material, or in an unlinked block that reversion misses. We therefore have domain-trained annotators audit a stratified sample of several hundred counterfactuals and report the label error rate per operation. Operations whose error rate is too high are dropped rather than patched.
The second stage trains Qwen3.5-9B, the open model that AutoSupervision fine-tunes, on the original training split plus counterfactuals. We sweep the mixing ratio between real and counterfactual examples. We apply response-only loss masking, meaning the loss is computed only on the model's answer tokens, because FirstPass showed that without it accuracy on long review contexts falls to 62%, below a majority guess.
The third stage is evaluation on failures the model was never shown as synthetic data. Accepted-paper corpora still contain real failed revisions: when a second-round reviewer writes that a first-round concern was not addressed, that is a naturally labeled unresolved instance. We extract these from the multi-round Nature Communications histories and from F1000 Research in NLPeer, which provides open reports and versioned articles, and we check them by hand. The resulting set measures real transfer, within and across publishers.
Baselines
The naive reference point is Qwen3.5-9B fine-tuned on the unaugmented AutoSupervision training split with the same masking and the same number of optimizer steps. Its per-class recall on "unresolved" shows how much the acceptance bias costs. AutoSupervision reports that this kind of fine-tuning substantially improves grounding for Qwen3.5-9B, but no reported numerical value was found on Lacuna. We will therefore reproduce it rather than quote it.
The strongest existing systems are frontier models as reported by AutoSupervision. GPT-5.5 reaches a verification score of 0.501 and an overall score of 0.637, and Claude-Opus-4.8 ties it at 0.637 overall. The agentic retrieval workflow, which retrieves manuscript blocks before judging, lifts GPT-4o-mini's verification from 0.266 to 0.403. We cite these numbers as published. We also rerun GPT-5.5 with agentic retrieval on our natural-failure set, because that set is new. Beating the frontier means that our 9B model's verification score on the AutoSupervision test split exceeds 0.501 under the benchmark's own scorer. It also means our model's unresolved-class recall on the natural-failure set is higher than the agentic frontier run's, with bootstrap confidence intervals that do not overlap.
Three further comparisons isolate the proposal's ingredient, each at matched example counts, training steps, and hyperparameter search budget:
- Reweighting: the same model trained with unresolved and partial examples upweighted to the counterfactual class ratio. This tests whether new examples matter or only class balance does.
- Generated failures: an LLM writes a failed revision for each concern, and we train on as many generated examples as we have counterfactuals. This tests recombined real text against generated text.
- Diff heuristic: a model-free rule that labels a concern resolved if and only if any aligned block changed. It measures how much of the task reduces to detecting change.
Finally, we ablate each counterfactual operation, and we separately ablate whether the response letter is held fixed or also edited.
Why not just…?
Why not just reweight the rare "unresolved" class? Reweighting repeats the few real failures, and those failures do not show the model the specific contrast between a letter that claims a fix and a manuscript that lacks it. If the reweighted model matches ours on the natural-failure set, the claim fails. We take that as a live possibility, and this comparison is the first decisive check.
Why not just have an LLM generate failed revisions? It is the obvious generative augmentation, and it may help. But generated revisions tend to differ from real ones in length and style, which gives the verifier an artifact to key on. The same form-over-substance bias documented in Articulate Intuition or Genuine Analysis? would reward that shortcut. We test this directly: if generated failures inflate in-distribution scores but lose to reversion on natural failures, the artifact story is supported. If they win, we will say so and switch to them.
Why not just use agentic retrieval at inference time? Retrieval is AutoSupervision's own remedy, and it clearly helps (0.266 to 0.403 for GPT-4o-mini). But retrieval supplies evidence without teaching the model to distrust the letter when the evidence is missing. Our method is complementary, and we will also evaluate our fine-tuned model inside the same retrieval loop.
Why not just check whether the text changed? This is the most serious threat. If reversion were the only operation, a model could learn exactly this rule. The diff heuristic baseline and the misattribution operation exist to settle it. A model that learned only change detection will fail on misattributed examples and on natural failures where authors edited the wrong thing.
Proposed Experiments
The decisive checks are four. First, verification score on the AutoSupervision test split, compared against the published 0.501 for GPT-5.5. Second, unresolved and partial recall on the hand-checked natural-failure set, where our model must beat reweighting, generated failures, and the diff heuristic. Third, a letter-swap probe: holding each manuscript fixed, we substitute a response letter that claims or denies the fix. A robust verifier's prediction should move with the manuscript, not the letter, and we report the rate at which predictions flip with the letter alone. Fourth, grounding accuracy on counterfactuals, checking whether the model points to the reverted block or correctly reports that no evidence exists.
Transfer from Nature Communications to F1000 tests whether the method generalizes beyond one journal's conventions. The audit error rates bound how much any gain could be an artifact of noisy constructed labels. If misattribution produces no improvement beyond reversion, we will conclude that the models were mostly learning to detect text changes, which is itself a useful negative result for anyone building revision checkers.
References
- AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification
- FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes
- Cut to the Mix: Simple Data Augmentation Outperforms Elaborate Ones in Limited Organ Segmentation Datasets
- A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability
- Fact or Fiction: Verifying Scientific Claims
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- NLPeer: A Unified Resource for the Computational Study of Peer Review