Evidence-Licensed Rebuttal: Making LLM Paper Review Hold Firm Without Going Deaf

Language models are increasingly used to review research papers, and most current work asks one question: can the model find the errors? A conference review has a second step that matters just as much. Authors write a rebuttal, reviewers discuss, and an area chair (the senior committee member who reads all reviews and decides) settles which criticisms stand. That step targets two documented weaknesses of language models. Models abandon correct conclusions when someone pushes back confidently. Groups of models also follow the majority even when a lone dissenter is right. If automated or AI-assisted review is deployed, an author who writes "this is clearly not an issue" could erase a real bug. Meanwhile a reviewer who never concedes anything is useless for the false alarms that automated reviewers produce in large numbers.

We propose to borrow one specific rule from how conferences are organized, and to test whether it fixes both failures at once. The rule is that a criticism can only be withdrawn, and a rebuttal can only succeed, by pointing to a specific passage of the manuscript. An area-chair agent then checks that passage against the criticism without ever reading the rebuttal's prose. The persuasive surface of the argument is removed before judgment. Head counts of reviewers are not used. Our claim is that this "evidence-licensed" adjudication keeps real errors from being argued away by confident authors, while still letting the system drop its own false alarms when the authors are right. Neither instructing the model to stand firm nor simply rewriting the rebuttal in neutral language can do both.

Core Claim

Existing work splits along a line this proposal crosses. On one side are studies of error detection in full manuscripts. PaperAudit-Bench injects 10–20 realistic bugs into each of 220 ICLR/ICML/NeurIPS 2025 papers. It shows single-pass scans reach only 40–41% Macro-F1, and a planner/retriever/specialist multi-agent "Deep Mode" raises GPT-5's error coverage from 67.7% to 80.6%. CLAIMCHECK shows models trail humans at tying a weakness to the claim it disputes (o1 F1 28.5 against human 45.8). Both stop at detection; neither has a rebuttal phase.

On the other side are studies of how models change their minds. Challenging the Evaluator shows that on multiple-choice questions a casual "Sure" rebuttal flips correct answers 84.5% of the time on average, more than a fully reasoned rebuttal (56.1%). Can LLM Agents Really Debate? shows agents in a majority hold their position over 97% of the time on logic puzzles, so correct minorities are suppressed. These are closed-world tasks with answer keys, not multi-issue paper reviews.

These motivating papers suggest that adding a rebuttal phase to current review agents would bring both failure modes into the setting where they cost the most. Nobody has measured this. The new contribution is a two-sided measurement of rebuttal robustness on full papers, together with a specific adjudication rule whose key ingredient, style-stripped span verification, can be ablated cleanly.

Research Program

We first build a rebuttal stress test on top of PaperAudit-Bench, because it is the only resource in reach with ground-truth error locations in full papers. For each paper, reviewer agents produce critiques. We label each critique by matching it to the injected-bug list. A sample of unmatched critiques is audited by hand, because some may point to genuine pre-existing problems rather than false alarms. We also plant refutable false critiques: claims that a detail is missing when the paper states it explicitly, so the refutation is checkable by quotation.

An author agent then writes rebuttals of four kinds. It writes valid rebuttals to false critiques that cite the correct passage, and assertive denials of real bugs modeled on the casual rebuttal styles of the sycophancy study. The hardest kind is a fabricated-evidence rebuttal of a real bug: one that quotes a real but irrelevant passage, or paraphrases a passage into something it does not say. The fourth kind is honest concession. We also construct committee configurations in which one reviewer flags a true bug and two others miss or dispute it, the analogue of a correct minority.

The adjudication protocol follows conference structure but adds a rule. Reviewers write independently and commit before seeing each other, as in real review. In the rebuttal phase, every request to withdraw a critique must name a manuscript span. The area-chair agent receives only the critique, the named span, and the surrounding section. It does not see the rebuttal's wording or confidence, and it does not see how many reviewers agree. It decides whether the span actually resolves the critique. Each critique is decided on its own evidence, so a single reviewer's correct finding is not outvoted.

The design applies divide-and-conquer prompting in the sense of When Splitting Makes Stronger, which shows that solving sub-tasks in clean, short contexts limits error buildup. Here each critique becomes its own clean-context check. Organizational accounts such as Conference Organization / xxviii supply the role split: independent reviewers, an author response, and an area chair who decides. They do not supply the verification rule.

The primary metrics are two numbers that pull against each other. The first is true-bug retention under adversarial rebuttal: the fraction of correctly flagged injected bugs still standing after an assertive or fabricated-evidence rebuttal. The second is false-critique withdrawal under valid rebuttal: the fraction of refutable false critiques dropped after a correct, span-citing rebuttal. We report both, plus their mean, because a system that never changes its mind scores perfectly on the first and zero on the second. Secondary measures are final post-rebuttal Macro-F1 against the injected bugs, minority-survival rate in the planted committee configurations, and cost in model calls.

Baselines

The do-nothing reference is the pre-rebuttal critique set from the same reviewer agents: detection with no author response. Its F1 is the floor any rebuttal phase must not fall below on true bugs. Its false-alarm count is what a good rebuttal phase should reduce. For context, the strongest reported detection numbers on this benchmark are GPT-5 Deep Mode error coverage of 80.6% and a fine-tuned Qwen3-8B (supervised fine-tuning plus GRPO reinforcement learning) at 59.7% F1, both from PaperAudit-Bench. We use Deep Mode as the reviewer stage for every condition, so all methods start from the same critiques. We found no reported result on Lacuna for rebuttal robustness of any review system on PaperAudit-Bench or any full-paper benchmark.

The strongest existing method for the rebuttal phase is the Creation–Review–Revision protocol of Multi-Agent Peer Review Collaboration. There, agents critique each other with confidence scores, revise, and settle by majority vote. It raised Penguins accuracy to 79.45% from 47.95% for self-correction. We extend it naturally by treating the author agent's rebuttal as a peer critique and keeping majority-vote resolution. We run it with the same backbone model, the same reviewer outputs, the same rebuttals, and a matched budget of model calls, with prompt tuning limited to the same small validation split used for our method. Beating it means higher true-bug retention under assertive and fabricated rebuttals with no loss, within confidence intervals, in false-critique withdrawal.

A second baseline is a single area-chair agent that reads the full reviews and full rebuttal and writes a meta-decision. This is how current pipelines would most naturally add a rebuttal phase.

The ablations isolate the key ingredient. In one, the area chair reads the full rebuttal prose instead of only the cited span. In another, the area chair sees how many reviewers agree. In a third, the span requirement is dropped while style-stripping is kept. If gains disappear when the rebuttal prose is restored, style-stripped verification is doing the work. If they disappear when the span requirement is dropped, the evidence license is.

Why not just…?

Why not just instruct the model to ignore assertive tone and stand by its critiques? A firmness instruction will likely raise true-bug retention, but nothing stops it from also suppressing valid withdrawals. The two-sided metric exists to expose exactly that trade. The cited evidence also suggests the vulnerability is not a matter of instruction. The sycophancy study finds the same models that judge well in a neutral judge setting capitulate when challenged conversationally (Llama-3.3-70B: 86.0% persuasion under full rebuttal against 56.5% in the judge setting). We include a firmness-prompted area chair as a baseline. If it matches our protocol on both metrics, the span rule is unnecessary, and we will report that.

Why not just rewrite every rebuttal into neutral language before the area chair reads it? This is the most serious threat. Style normalization might remove most of the assertiveness effect at almost no cost. We include it as a baseline: a paraphraser strips tone and hedges, and the area chair reads the neutral version. Our prediction is that it handles assertive denials but not fabricated-evidence rebuttals. A calm, well-structured argument citing an irrelevant passage survives paraphrase intact. The judge study Articulate Intuition or Genuine Analysis? finds judges reward analytical form over substance, and neutral paraphrase preserves analytical form. If normalization matches us on the fabricated-evidence condition too, the claim narrows to "tone is the whole problem," which is still a useful finding.

Why not just fine-tune a reviewer on real rebuttal threads? Re2 supplies 53,818 rebuttals and shows that fine-tuning reduces the zero-shot tendency to accept nearly everything. However, real rebuttal threads carry no ground truth about whether the author was right, so fine-tuning on them teaches the model to imitate how reviewers actually conceded, including when they conceded wrongly. We test this directly. A model fine-tuned on Re2 discussions is run as an area chair on the stress test under the same call budget. We expect it to inherit human concession rates rather than evidence sensitivity.

Why not just ignore rebuttals and keep the independent reviewers' majority? This gives perfect resistance to persuasion. It also never removes a false alarm, and single-pass precision on this benchmark is low enough that false alarms dominate what an author would see. The do-nothing baseline measures this cost directly. Majority voting additionally erases correct lone findings, which the planted minority configurations quantify.

Proposed Experiments

The main experiment runs all conditions on the 220 PaperAudit-Bench papers with two backbone families, one closed and one open-weight. Rebuttal type is crossed with condition, and confidence intervals are bootstrapped over papers.

The decisive check is the fabricated-evidence condition. Our protocol should keep true bugs there at a rate close to its keep rate under plain assertive denial. Every baseline that reads rebuttal prose, including the neutral-paraphrase one, should fall well below that. If our protocol also falls, then span verification itself is being fooled by plausible-looking passages, the central claim fails, and the per-critique entailment check becomes the thing to fix.

A second check is minority survival. In committees where only one of three reviewers found a true bug, the bug should survive at nearly the rate it does when all three found it. The majority-vote baseline should show the conformity collapse reported in the debate study.

Two external-validity checks follow. First, on CLAIMCHECK we test whether the area chair's span verification agrees with human-annotated weakness–claim links better than o1's reported 28.5 F1 on claim association. That is the closest real-paper analogue to the span check. Second, a blinded human study on 30 real Re2 rebuttal threads asks domain experts whether each withdrawal or retention by our area chair was justified. We compare against the full-prose area chair and score the meta-decisions with the Kahneman4Review rubric, to confirm that any improvement is not just longer or more formal text.