Verifying What Authors Say They Fixed: Hybrid Retrieval Across Manuscript Revisions

When a paper goes through a revision round, as at PoPETS or under PVLDB-style shepherding, the authors send back two things: a revised manuscript and a response letter. The letter says, point by point, what changed. "We added the ablation in Table 6." "The bound in Theorem 2 now holds for all $k$." "We corrected the baseline numbers." Reviewers usually read the letter and skim the new draft. Language models asked to help them tend to do the same thing, only more politely. The response letter is a dense, fluent, persuasive set of factual claims about a document, which is the kind of text language models fail to check. This proposal treats each claim in the letter as a fact-checking target. We would decompose the letter into single checkable statements and retrieve evidence for each one from both drafts and the text diff between them. Retrieval combines exact keyword matching with meaning-based matching, so that table numbers, equation symbols, and baseline values are matched exactly while paraphrased claims are still found. Most verdicts would come from cheap calibrated scores, and a strong language model would be called only on the cases those scores leave uncertain.

The motivation is concrete. Iterative review exists so that flaws get fixed rather than discarded (Balazinska et al.; Editors' Introduction, PoPETS 2022). That benefit depends on someone confirming that the fix actually happened. Automated auditing so far examines one draft at a time. Simulated multi-round review shows that agents anchor on first impressions and conform to each other instead of checking facts (AgentReview). The revision round is therefore the point where the process is supposed to be most rigorous, yet no automated system currently verifies it.

Core Claim

A response letter turns the open-ended problem "find the bugs in this paper" into a much easier problem: "does the evidence for this specific claim exist where the authors say it does?" We claim that a verifier built for this narrower problem will beat two kinds of system on detecting false or partial fix claims, at a fraction of the cost. The first is a frontier long-context model given both drafts and the letter. The second is a full re-audit of the revised draft. The verifier has three parts: claim decomposition, hybrid retrieval over a two-draft, section-aware index, and calibrated escalation. We further claim that exact lexical matching carries most of the gain. Revision claims are dominated by identifiers such as table numbers, symbols, metric values, and baseline names, and dense encoders compress these away.

Relation to Prior Work

The closest system is CiteGuard. It fuses keyword search (BM25) and scientific-paper embeddings (SPECTER2) with reciprocal rank fusion, a method that merges ranked lists by summing inverse ranks. It then scores candidates on several alignment features, calibrates those scores with isotonic regression, and escalates only uncertain cases to a small language model. On citation verification it reports citation-level F1 of 0.89 while escalating 18% of citations. We reuse this design almost exactly: CiteGuard supplies the method. Its target, however, is whether a reference exists and supports a sentence, which is bibliographic, single-pass, and single-document. Our target is whether a claim about a change between two versions of the same document is true. That requires indexing both drafts, using the diff as evidence, and checking numbers and symbols across sections.

Hybrid Retrieval for Hallucination Mitigation motivates the choice of hybrid over single-mode retrieval. On HaluBench it reports MAP@3 of 0.897 for its hybrid retriever, against 0.768 for dense-only and 0.724 for sparse-only, and a PubMedQA hallucination rate of 4% against 16% for sparse and 20% for dense. Its query-specificity switch, which up-weights keyword search for technical queries, is a heuristic we test directly. That paper works on one-shot question answering, not on documents that change.

PaperAudit-Bench both motivates and constrains the work. It shows that GPT-5 and Gemini 2.5 Pro reach only 40–41% Macro-F1 when scanning a full paper for injected bugs. Errors in the abstract are hardest to catch, and a multi-agent "Deep Mode" with an internal retriever raises GPT-5 coverage from 67.7% to 80.6%. It audits a single draft and never sees a response letter. Its injected-bug corpus, where clean and corrupted versions of the same paper both exist, is the raw material for our benchmark.

Reviewing Scientific Papers for Critical Problems and MMReview are also single-pass. They document large variation across model vendors (o3 finds 350 true problems against Claude's 149 on WITHDRARXIV-CHECK) and a bias toward longer papers. Both results argue against depending on whichever long-context model is current.

FActScore supplies the habit of splitting long text into atomic claims and checking each one. It also warns that precision drops on rare entities and later in long outputs, which is the regime response letters occupy. Sparse, Dense, and Attentional Representations explains why a single dense vector per long section will fail: the score gap between the right passage and its competitors shrinks as length grows. For that reason we index at the passage, table-cell, and equation level. Finally, AgentReview shows that rebuttals barely move agent reviewers. That result is what we test in our secondary experiment: whether grounded verdicts, rather than more discussion, change outcomes.

Research Program

The first step is building the benchmark, since none exists. Each PaperAudit paper comes in a clean original and a version with 10–20 injected bugs, and we treat the buggy version as round one. Round two is built by programmatically repairing a random subset of the bugs, restoring the clean text for those spans. Other bugs are repaired only partially, for example by fixing a table value but not the sentence that cites it. The rest are left untouched. In some papers we also inject revision regressions, meaning new inconsistencies introduced by the edit itself, such as a new table row that contradicts the revised abstract.

A language model then writes a response letter for each paper that claims every bug was addressed. The letter is written in the style of real responses, and some claims are paraphrased away from the manuscript's wording on purpose. Ground-truth labels for each claim (fixed, partially fixed, not fixed, or fixed but regressed elsewhere) follow from the construction.

Synthetic letters may be easier than real ones, so we would add a human-annotated set. It would consist of real revision pairs with author responses, taken from public OpenReview histories of ICLR submissions, the same source AgentReview drew on. Domain experts would label each response claim. We report results on the two sets separately.

The verifier has four stages:

  1. Claim decomposition. A language model splits the letter into atomic claims and extracts the identifiers each one names: tables, equations, numbers, and section references.
  2. Indexing. Both drafts are indexed at the granularity of paragraphs, table cells, equations, and captions, and each unit is tagged with whether it falls inside a changed region of the diff.
  3. Retrieval. For each claim, keyword search and a scientific dense encoder retrieve candidate evidence from round two, round one, and the diff hunks, and the results are merged by rank fusion.
  4. Scoring and escalation. Alignment features are computed: exact numeric agreement between claim and evidence, whether the cited identifier exists in round two, whether the evidence changed relative to round one, dense similarity, and the consistency of the same number wherever it appears in round two. Isotonic regression turns these features into a calibrated probability that the claim holds. Claims with intermediate probabilities go to a strong reasoning model, which sees only the retrieved evidence.

The cross-occurrence consistency feature targets PaperAudit's abstract blind spot. It checks the abstract and introduction against whatever the revised tables now say.

As a secondary study, we feed the verifier's claim-level verdicts into an AgentReview-style deliberation between reviewers and an area chair. We then measure whether grounded verdicts move scores toward the correct accept or revise outcome more than ungrounded rebuttals do. This tests whether the conformity effect AgentReview reports is driven by missing facts.

Baselines

The naive reference point is to accept every claim in the response letter as true. Its precision equals the fraction of genuinely fixed claims in the benchmark, and its recall on false claims is zero. Any method must clearly beat it on false-claim F1.

The strongest competitor is a frontier long-context reasoning model, the same model we use for escalation, given the full round-one draft, the round-two draft, and the letter. It is asked for the same claim-level verdicts. This is the strongest baseline because it sees strictly more text than our verifier and needs no engineering. The file has no reported result for this revision-verification task. The closest reported figures come from single-draft bug finding: 40–41% Macro-F1 in PaperAudit's Fast Mode for GPT-5 and Gemini 2.5 Pro. To give this baseline its best shot, we tune its prompt with the same budget of development-set iterations we spend on our verifier, and we allow it chain-of-thought.

The second strong competitor is PaperAudit's Deep Mode re-run from scratch on the round-two draft. We map its findings onto claims: a residual bug that it finds counts as detecting the corresponding false fix claim. Its reported coverage of 80.6% for GPT-5 on single drafts is the number to respect. It ignores the letter, so it pays for a full audit to find what the letter localizes.

The third competitor is a diff-plus-model pipeline. A standard text diff isolates changed hunks, and the same model judges each claim against those hunks only.

Ablations isolate our key ingredients. We run dense-only and keyword-only retrieval, and we run the Mala et al. specificity switch in place of learned feature weights. We remove round one and the diff from the index, so the verifier sees only the revised draft. We remove calibration, either escalating every claim or never escalating.

All systems share the same escalation model and the same development split. We report tokens consumed and wall-clock time for each. Beating the long-context baseline means higher false-claim F1 on both the synthetic and the human-annotated sets at no more than a quarter of its token cost. Matching it at lower cost is still useful, but it is a weaker result.

Why not just…?

Why not just give a long-context model both drafts and the letter? This is the real threat, and the proposal is built to settle it rather than assume it away. The evidence against it is indirect. On single drafts, frontier models miss most distributed inconsistencies (40–41% Macro-F1 in PaperAudit Fast Mode). They are weakest on abstract-level claims and vary widely across vendors (350 against 149 true positives on WITHDRARXIV-CHECK). FActScore shows precision decaying on rare entities, and revision claims are full of rare identifiers. Still, a response letter tells the model where to look, which may close much of that gap. If the long-context baseline matches our verifier on the human-annotated set, the honest conclusion is that retrieval buys only cost savings, and we would report it that way.

Why not just diff the drafts and have a model read the changed parts? A diff catches claims of change where nothing changed, so this baseline will be strong on those cases. It cannot see two kinds of failure. One is a fix that changed a table but left a stale number in the abstract or appendix, which shows up as an inconsistency between a changed region and an unchanged one. The other is a claim whose supporting evidence was already present in round one. The partial-fix and regression labels in our benchmark exist to measure exactly these cases. The ablation that removes round one and the diff, together with the diff-only baseline, will show how much the retrieval over unchanged text adds.

Why not just re-audit the revised paper from scratch? Re-auditing is the right tool when there is no letter. When there is one, it wastes the localization the authors provide and inherits the weaknesses of single-draft auditing. It also cannot judge partial fixes as partial, because it has no record of what was promised. We compare at equal token budget. Our prediction is that letter-targeted verification reaches higher recall on residual bugs among claimed fixes. A single-draft auditor, by contrast, should keep its advantage on bugs the letter never mentions, and we report those separately.

Why not just use dense retrieval, which is the default in most retrieval-augmented pipelines? Revision claims hinge on strings like "Table 6", "$\lambda = 0.1$", and "81.3". Dense encoders blur these, and long sections make it worse, because the score gap shrinks with length (Luan et al.). Hybrid retrieval cut the PubMedQA hallucination rate from 20% (dense) to 4% (Mala et al.). The dense-only ablation measures whether the same holds here.

Proposed Experiments and Decisive Checks

The primary metric is claim-level F1 for detecting false or partial fix claims. We also report residual-bug recall broken down by PaperAudit's eight error dimensions, regression detection rate, escalation rate, tokens per paper, and latency. We report the synthetic and human-annotated sets separately.

Four checks decide the project:

  1. Does the verifier beat the long-context baseline on the human-annotated set at a fraction of its cost?
  2. Does the keyword-only ablation lose more than the dense-only ablation on numeric and identifier claims? If so, that confirms where the gain comes from.
  3. Does detection of abstract-level stale claims rise well above PaperAudit's reported weakness on abstracts?
  4. In the deliberation study, do grounded verdicts reduce the share of decisions flipped by a single biased reviewer below the 37.1% AgentReview reports for its ungrounded setup?

A negative answer on the first check, combined with a positive one on cost, would narrow the claim to efficiency. A negative answer on the second would mean the hybrid index is not the key ingredient.