When an editorial board decides on a paper, information moves upward through tiers. Several reviewers each write free text. An area chair, sometimes called an associate editor, condenses that text. A senior editor then decides. Each handoff is a chance for substance to be lost and style to take over. That risk is no longer hypothetical once language models sit in these seats. Articulate Intuition or Genuine Analysis? shows that an LLM judging reviews rewards fluent, analytical-looking prose, and that review length explains about 74% of the apparent quality advantage of AI reviewers over humans. We propose to change what is passed between tiers. Instead of free-text summaries, each tier receives a claim ledger. A ledger is built from predicate-argument structure: a sentence reduced to "who did what to whom" tuples, such as evaluate(method, dataset-X). The manuscript's claims are extracted as tuples. Every reviewer critique and every author rebuttal sentence is then attached to the specific claim it targets. The upper tiers decide from that ledger.
The core idea fits in two sentences. Upper tiers of an editorial hierarchy should see which manuscript claims were challenged, by how many reviewers independently, and whether the rebuttal answered them, rather than how eloquently the challenges were phrased. Predicate-argument tuples give each claim a stable identity across differently worded reviews, so agreement and resolution can be counted rather than guessed.
Core Claim
An automated meta-review tier that reads a claim ledger anchored in the manuscript's predicate-argument tuples will meet two conditions, where a long-context LLM reading the raw paper and reviews will not. Its decisions will be largely invariant to content-free stylistic inflation of reviews, such as padding, hedging and fluent restatement. They will also stay sensitive to deleting a substantive critique. The claim-level granularity matters most where paragraph-level anchoring fails: dense passages that make several distinct claims, and critiques worded with little lexical overlap with the text they target.
The relation to the closest work is as follows. Argument Mining for Understanding Peer Reviews segments reviews into typed propositions (Evaluation, Request, Fact and others). It never links them to the manuscript and leaves relations between propositions to future work. APE pairs review arguments with rebuttal arguments but cannot see the manuscript. It also models only the two-party reviewer–author exchange, not the meta-reviewer who judges whether a rebuttal succeeded. Kahneman4Review diagnoses the form-over-function failure but evaluates reviews in isolation, without the paper. ResearchTown simulates reviewing as text passing between agents, with no editorial tiers and no structured claims. Our contribution is the inter-tier interface itself, and a test of whether that interface makes decisions depend on substance.
Research Program
The first component is manuscript claim extraction. We parse each submission's abstract, introduction, method and results sections into predicate-argument tuples. The method follows the dependency-based event-tuple extraction of U-CREAT, which uses such tuples to strip legal boilerplate and keep factual content. It also follows the dependency-relation patterns of DepChecker, which extracts attribute-value hypotheses and checks them against a catalog. Both papers supply method, not motivation. We add an LLM-based extractor that emits the same tuple schema and compare the two. PARMA warns that pipelined extractors inherit parser errors on domain text. Scientific prose is exactly such text, so we measure extraction quality directly on a gold subset rather than assuming it.
The second component is critique and rebuttal anchoring. Each review proposition, typed with the scheme of the argument-mining paper, is aligned to zero or more manuscript tuples. "Zero" is a meaningful outcome: it flags critiques of claims the paper never makes. Alignment follows PARMA's design, which aligns predicates and arguments separately rather than matching surface lemmas. This choice is constrained by evidence. APE reports only 9.9% vocabulary overlap between paired review and rebuttal arguments, and PARMA reports that lemma matching collapses under realistic rewording. Rebuttal sentences are anchored the same way. A critique and a response that land on the same tuple form a candidate resolution pair, and a verifier labels it addressed, partially addressed or unaddressed.
The third component is the ledger and the tier protocol. The ledger groups anchored items by manuscript claim. Each entry records the critique type, the number of distinct reviewers raising it, and its resolution status. The meta-review tier reads the ledger and writes a decision and rationale. The senior tier sees only the ledger plus the meta-review, never the raw reviews, which mirrors how vertical oversight limits what downstream agents can be swayed by. As a secondary question we follow Voting or Consensus?, which found that consensus filters factual errors better and voting handles open-ended reasoning better. On that basis we test whether Fact-type ledger entries should be settled by agreement-plus-verification while Evaluation-type entries are aggregated by vote. The secondary question does not carry the main claim.
Data
We use the PeerRead ICLR portion, which contains drafts, reviews and accept/reject outcomes. We also use the Review-Rebuttal dataset from APE, linking each submission to a manuscript where the submission can be matched. We report the matched fraction rather than assuming it is large. The AMPERE proposition annotations (400 ICLR 2018 reviews, 10,386 typed propositions, κ = 0.93) supply typed critiques. We will add our own gold annotation of critique-to-claim links on a few hundred of these reviews, which requires retrieving the corresponding manuscripts. Kahneman4Review's 3,563 rated reviews, including AI-written ones, serve as the stress set for length and style effects.
Baselines
The naive reference points are a majority-class predictor and PeerRead's surface-feature acceptance classifier. PeerRead reports 65.3% accuracy on ICLR using coarse features such as length and appendix presence. That number is a warning as much as a baseline. If the ledger system does not clearly beat it on accuracy, any gains it shows must come from invariance rather than prediction.
The strongest competitor is a long-context LLM acting as area chair. It receives the full manuscript, all reviews and the rebuttal in one prompt, with the same model, decoding settings and prompt-tuning budget as the ledger-reading tier. It is strongest because it has strictly more information than the ledger and needs no extraction pipeline. No reported result was found on Lacuna for a long-context LLM's meta-review decision accuracy on PeerRead or the Review-Rebuttal data, so we establish this number ourselves. Beating it means matching its decision accuracy within noise while showing a significantly lower decision-flip rate under style inflation and an equal or higher flip rate under substantive deletion.
A structured but unanchored baseline has the same LLM first summarize each review into a list of critiques and then decide from the lists. This isolates whether gains come from structure alone or from anchoring to the manuscript.
For anchoring itself, two prior systems serve as reference points. The first is APE's MT-H-LSTM-CRF, which reports 29.81 F1 on argument pair extraction, against about 20.23 for a pipeline baseline. The second is PARMA, which reports 59.2 F1 on MTC against 42.1 for lemma matching, for cross-document predicate-argument alignment. Our critique-to-claim linker is compared against PARMA run on our gold links, a lemma baseline, and dense-embedding nearest neighbor, all on the same splits.
The key ablation replaces predicate-argument anchors with paragraph-level anchors and keeps everything else fixed. This directly tests whether claim granularity, rather than mere manuscript grounding, does the work.
Why not just…?
Why not just give a long-context LLM the paper and reviews and let it decide? This is the real threat, and it is the main comparison rather than something we argue away. The evidence that it falls short is indirect. Kahneman4Review finds that a frontier judge collapses nine evaluation dimensions into one "analytical-trace" factor driven by fluency. That finding concerns judging reviews, however, not judging papers given reviews. The style-inflation experiment below settles the question. If the end-to-end model's decisions do not move when reviews are padded, the ledger's main justification disappears and we will report that.
Why not just anchor critiques to paragraphs or retrieved passages instead of extracting tuples? This alternative may work as well on average, so it is built in as the decisive ablation rather than dismissed. Our prediction is narrow. Tuples should win where one paragraph carries several claims, because two reviewers critiquing different claims in one paragraph would be wrongly counted as agreeing. Tuples should also win where critique wording diverges from the manuscript, given APE's 9.9% overlap figure. If paragraph anchoring matches tuple anchoring even in high-claim-density paragraphs, we will conclude that grounding, not predicate-argument structure, is what matters.
Why not just have the LLM summarize reviews into critique lists without touching the manuscript? Summaries remove some verbosity, but they cannot tell a critique of a real claim from a critique of a misread one. They also cannot verify that a rebuttal addressed the same claim the reviewer raised. The structured-but-unanchored baseline measures exactly this. The misread-critique test below targets the capability that only manuscript anchoring provides.
Why not just use flat voting among reviewer scores? Flat score averaging is the de facto human-free aggregator, and Voting or Consensus? shows that no single flat protocol suits both factual and evaluative content. We include score averaging as a baseline. It is, however, trivially invariant to style in the text, so we judge it on accuracy and on whether it responds to unresolved factual critiques.
Proposed Experiments
The decisive check is a paired perturbation study, modeled on the function-probe pilot in Kahneman4Review (33 of 35 pairs). For each held-out submission we create two edits. A style-inflated version doubles review length with fluent, content-free elaboration, verified by annotators to add no new critique. A content-deleted version removes one critique raised by at least two reviewers and left unaddressed in the rebuttal. A good aggregator should rarely flip its decision under the first edit and should flip, or lower its confidence, under the second. We report both flip rates with bootstrap intervals for every system. The proposal succeeds if the ledger system lowers style sensitivity relative to the long-context baseline without lowering content sensitivity.
Second, we measure critique-to-claim linking F1 on the gold set. Results are stratified by claims per paragraph and by lexical overlap between critique and target, which is where the tuple-versus-paragraph ablation should separate. We also report how often the extractor misses the targeted claim entirely, since extraction failure caps everything downstream.
Third, we run a misread-critique test. We insert critiques that refer to claims absent from the manuscript, for example "the paper only evaluates on dataset X" when it evaluates on several datasets. We then measure whether each system discounts them.
Fourth, we report decision accuracy against real ICLR outcomes on PeerRead. We treat it as a sanity check against the 65.3% surface-feature baseline, not as the headline, because real decisions are themselves noisy.
Finally, on Kahneman4Review's agentic reviews we test whether the length advantage the paper identified still moves meta-decisions once reviews pass through the ledger. That is the most direct evidence on whether the interface removes the form-over-function channel.
References
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- Argument Mining for Understanding Peer Reviews
- APE: Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning
- ResearchTown: Simulator of Human Research Community
- U-CREAT: Unsupervised Case Retrieval using Events extrAcTion
- Fact Checking Machine Generated Text with Dependency Trees
- PARMA: A Predicate Argument Aligner
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications