Calibrated Hand-offs Between LLM Reviewers and the Editor

Automated peer-review systems built from several language-model agents tie two different jobs together. One job is judging a paper: spotting flaws, checking evidence, scoring soundness or clarity. The other is deciding what to do with those judgments: accept, reject, or escalate. Today the decision rule is written directly against the reviewers' raw scores, so it silently breaks whenever the reviewer model, its prompt, or the submission pool changes. This proposal separates the two jobs the way a journal does, and the way software engineers separate business rules from the code that produces their inputs. Reviewer agents for each concern (soundness, experiments, clarity, novelty) produce scores and quoted evidence. A thin calibration layer turns each concern's score into a probability fit on a small labeled anchor set of papers. An editor-in-chief tier is a fixed, human-readable policy written over those probabilities, for example "accept if the probability the paper is accept-worthy exceeds 0.6 and no verified soundness flaw has probability above 0.3."

The claim is narrow and testable. If reviewer outputs are calibrated per concern before they reach the editor, an editorial policy written once keeps its meaning when the reviewer backbone is swapped, when prompts are revised, and when the quality mix of submissions drifts. Recalibrating needs only a few dozen labeled papers and no change to the policy. This is the multi-agent analogue of Decoupling Decision-Making in Fraud Prevention through Classifier Calibration, which keeps fraud-alert business thresholds fixed while the classifier underneath is retrained. That paper worked on one tabular score. Here the scores are several, come from language-model judges, and feed a compound rule.

Where this sits

Two pieces of prior work supply the reviewer side. MARG splits review into worker agents that hold document chunks, persona experts, a leader, and a fact-checking group that prunes hallucinated criticisms. On the ARIES corpus it produces 3.7 "good" comments per paper against 1.7 for single-agent GPT-4. Its pipeline is hard-coded, though, and it has no editorial tier that sets thresholds across experts. MAS-Architect separates a declarative topology (who talks to whom) from node implementation through a shared state schema, reaching 78.7% average accuracy on reasoning benchmarks. Its separation is between graph and nodes. It says nothing about the interface between evaluators and decision-makers, which is where this proposal makes its cut.

Two further papers motivate the gap. Why Do Multi-Agent LLM Systems Fail? attributes 23.5% of failures across 1,642 traces to verification. It also finds that giving one agent sole termination authority improves ChatDev success by 9.4%, and that a separate high-level verification step adds 15.6%. Both results support an explicit top tier with a well-defined input. Embodied LLM Agents Learn to Cooperate in Organized Teams shows that designated leaders cut completion time (9.76% for three GPT-3.5 agents) without raising communication cost. It also warns that agent behavior is hard to bound, which argues for an editor whose rule is inspectable rather than another free-form language-model judgment.

DyLAN constrains the design from the other side. It prunes agents mid-task using importance scores and beats debate with 36.6% fewer API calls. Calibrated per-concern probabilities give a principled pruning signal: a concern reviewer whose calibrated output is uninformative on the anchor set can be dropped without touching the policy. ResearchTown simulates review as community message-passing and scores reviews by semantic similarity (0.49 on review writing). It never models the decision step, which is the step this proposal targets.

Research Program

The system has four layers bridged by a fixed state schema.

The first layer is the concern reviewers. Following MARG, each one sees the paper through chunk workers and emits a raw score in [0,1] together with quoted spans.

The second layer is a verification tier. It reuses MARG's refinement idea to keep or discard each cited flaw, and outputs a verified-flaw indicator per concern.

The third layer is the calibration layer. For each concern $c$ it fits a monotone map $g_c$ from raw score $s_c$ to $p_c = g_c(s_c)$, the probability that human reviewers flagged concern $c$ (or, for the aggregate, that the paper was accepted), using isotonic regression or Platt scaling on an anchor set of $n$ labeled papers.

The fourth layer is the editor policy $\pi(p_1,\dots,p_k)$, written once and frozen.

The scientific question is whether $\pi$'s decisions stay accurate and stable when everything below the calibration layer changes. A second question is how small $n$ can be. Calibration error from isotonic fits shrinks roughly as $n^{-1/3}$ (the standard rate for monotone regression), so we expect diminishing returns past a few dozen to a hundred anchors per concern and will measure where that happens.

Baselines

The do-nothing reference is a single long-context model given the full paper and the venue's reviewing guidelines, asked directly for accept or reject. It fixes the floor and checks whether decomposition matters for decisions at all, not just for comment quality.

The strongest existing system is a MARG-style multi-agent reviewer whose aggregate raw score is thresholded at a value tuned once on the anchor set and then held fixed across swaps. This is how such systems are deployed today. For comment quality, the reported figure to match is MARG-S's 3.7 good comments per paper on ARIES, versus 1.7 for single-agent GPT-4 (MARG). For accept/reject agreement with venue decisions under reviewer swaps, there is no reported result found on Lacuna. All arms share the same backbone models, the same anchor papers, the same token budget per paper, and the same number of tuning runs.

The third baseline is per-model quantile normalization. Raw scores are rank-transformed within each batch and the policy thresholds the ranks, which fixes the acceptance rate by construction and needs no labels.

The fourth is an oracle that retunes the raw threshold after every swap using the same $n$ anchors. It is an upper bound for single-threshold policies.

Two ablations isolate the key ingredient. One calibrates only the aggregate score rather than each concern. The other removes the verification tier.

Beating the field means three things. Decision accuracy after a backbone swap must stay within noise of the oracle without editing the policy. Accuracy must exceed the fixed-threshold MARG-style system by a margin confirmed by paired bootstrap. On quality-shifted batches, acceptance-rate error must be lower than quantile normalization's.

Why not just…?

Why not just ask one strong model for the decision? MARG's evidence is that single-agent reviews are mostly generic: 60% of comments versus 29% for MARG-S. That result concerns comment quality, not decisions. A single model might still guess accept/reject well, and the do-nothing baseline measures exactly this. If it matches the calibrated system on accuracy and stability, the decomposition is not earning its cost (MARG reports over 1.2 million input tokens per review), and we will report that.

Why not just rank-normalize scores? This is a real threat, because it makes any reviewer swap harmless for the acceptance rate at zero labeling cost. Its weakness is that it assumes every batch has the same quality mix. It will accept the top 25% of a weak batch as readily as of a strong one. The quality-shifted-batch experiment exists to settle this. If submission pools in practice do not drift enough to matter, quantile normalization wins and the proposal should say so.

Why not just retune the threshold after each swap? For a policy with a single threshold, retuning on $n$ anchors is nearly equivalent to isotonic calibration. We concede that case. The separation pays off only when the editor's policy is compound (per-concern vetoes, escalation bands) or itself changes, for instance a new target acceptance rate. In those cases calibrated probabilities let the policy be rewritten without relabeling, whereas raw thresholds must be searched jointly. Experiments therefore report single-threshold and compound policies separately, and claim an advantage only where it appears.

Proposed Experiments

Papers come from ReviewBench in ulab-ai/research-bench: 200 review tasks from NeurIPS 2024 and ICLR 2024. Each is joined to its public OpenReview decision and reviewer scores. This join is our own addition; the file does not state that the benchmark includes decisions. Per-concern labels come from human reviewers' soundness and presentation scores. ARIES is used to check that adding calibration does not degrade comment specificity relative to MARG-S.

The first decisive check is the backbone swap. We freeze a compound policy and replace the reviewer model across three or more model families and two prompt variants. We then measure decision accuracy against venue decisions, the rate at which decisions flip relative to the original backbone, and per-concern expected calibration error.

The second is the drift check. We resample batches whose share of accepted papers ranges from 10% to 50%, and measure the gap between the system's acceptance rate and the true rate in each batch. This is where calibration and quantile normalization should part ways.

The third is an anchor-size sweep from $n=10$ to $n=150$, to locate where calibrated accuracy reaches the oracle.

Finally, every run's traces will be labeled with the failure categories of Why Do Multi-Agent LLM Systems Fail?, to test whether a calibrated, rule-based editor reduces verification and termination failures relative to a free-form leader agent.