Peer review increasingly runs on partial reading. Human reviewers under load skim abstracts and figures, and LLM reviewers compress long papers or lose track of their middle sections. Most proposals to fix this make the single reader read better. This note asks a different question: what should the editor do when every reader below them reads only part of the paper? Our hypothesis is that skimming hurts editorial decisions mainly by making reviewers' mistakes correlated. Every skimmer reads the same salient parts (abstract, introduction, headline table), so they are all misled by the same overclaim. Adding reviewers then stops helping, and an editor who only reads their reports inherits the shared blind spot.
We propose to test, under a fixed total reading budget, whether two concrete editorial actions recover decision quality that averaging cannot. The first is coverage assignment: the associate editor gives each reviewer a different section of the paper to read deeply. The second is an evidence check: every criticism must quote the span it rests on, and the editor verifies that the quote exists and supports the claim before weighing it. If the hypothesis holds, a three-tier board of partial readers with these two rules should match or beat full-paper readers at equal cost. It should also resist the prestige and rebuttal manipulations that sway single LLM referees. If it fails, we learn that hierarchy adds nothing beyond better single-agent reading, which is also worth knowing.
Core Claim
The two directions meet at one mechanism. The cognitive-fragmentation literature describes readers who maximize throughput over depth. Examples include 2x playback, intro skipping and condensed summaries (Accelerated Fragmentation) and lookup-dominated reading, where only 21% of Wikipedia readers seek in-depth understanding (Why We Read Wikipedia). These studies are correlational or qualitative (Content Fragmentation Propagation in Short Videos, Short Videos and College Students' Cognitive Abilities). We use them only to motivate which reading policies to simulate, not as evidence about reviewers.
The editorial-structure literature, from formal tiered boards (IEEE Transactions on Information Theory publication information) to principal-agent analyses of long delegation chains (Principal-Agent Theory in Chinese Vocational Colleges), assumes that lower tiers pass up faithful signals. That assumption breaks when the lower tiers are skimmers.
The reason averaging fails is simple. Suppose each of $k$ reviewers has score error with variance $\sigma^2$ and pairwise correlation $\rho$. The error variance of their mean is
$$ \mathrm{Var}(\bar e) = \sigma^2\left(\rho + \frac{1-\rho}{k}\right), $$
which follows from summing the covariance matrix of the errors. More reviewers drive error down only to the floor $\rho\sigma^2$. Our claim is that skimming raises $\rho$, and that coverage assignment lowers it by giving reviewers disjoint evidence. Evidence checks attack a different failure: they let the editor discard criticisms that sound plausible but are ungrounded, without the editor reading the whole paper.
Research Program
We build a simulated three-tier board of LLM agents: reviewers, an associate editor and an editor-in-chief. The structure follows the five-phase pipeline of AgentReview, which showed that area-chair style alone shifts 30.6% of decisions and that a single biased reviewer shifts outcomes in 37.1% of cases. AgentReview's agents all read the full paper. We replace full reading with controlled reading policies.
The policies are matched by total tokens consumed across the board. "Skim" gives a reviewer the abstract, introduction, figure captions and conclusion. "Compressed" runs the whole paper through prompt compression to the same budget, in the spirit of the chunking in TreeReview. "Random-chunk" samples paragraphs uniformly. "Assigned" gives each reviewer a disjoint section chosen by the associate editor from the table of contents, plus the abstract.
Report format is crossed with reading policy. Free-form reports are compared against evidence-carrying reports, in which each weakness is tied to a verbatim quote and a target claim, mirroring the claim-association task of CLAIMCHECK. The associate editor's verification step runs two checks. It does an exact or near-exact string match of each quote against the paper. It also makes a narrow entailment judgment: does the quoted span bear on the stated claim? Weaknesses that fail either check are down-weighted before the meta-review. The editor-in-chief sees only the meta-review and the verified evidence, which makes this tier the extreme case of a fragmented reader.
Three quantities are measured throughout. The first is decision quality against real outcomes. The second is the empirical reviewer-error correlation $\rho$, estimated from residuals of reviewer scores against the human mean score. The third is robustness to the manipulations catalogued by When AI reviews science, which found that single frontier referees gain +0.53 points after evidence-free rebuttals (89% of papers scored higher) and lose 0.72 points under low-prestige framing. The prediction specific to our account is that evidence checks should blunt rebuttal sycophancy. A rebuttal that cites no span gives the verifier nothing to accept.
Baselines
The naive reference point is a single reviewer agent that skims under the per-reviewer budget and whose score is the decision. Every board configuration must beat it, or the study is moot.
The second baseline is independent averaging. We run $k$ skimming reviewers with no editor logic and threshold their mean score. This is the classic "more eyes" answer and the direct test of the correlated-error claim. We run $k$ from 1 to 6 at matched total tokens and fit the floor in the equation above. Beating it means a lower decision error at the same total budget, plus a measurably lower $\rho$ for assigned reading.
The strongest existing single-agent method is TreeReview. It decomposes a paper into a tree of sub-questions, answers leaves over compressed 1,024-token chunks, and reports specificity +12.27%, comprehensiveness +11.22% and an 80.2% token saving over a multi-agent baseline, judged by an LLM. The file reports no accept/reject accuracy for TreeReview, so we will run it ourselves on the same papers, with the same backbone model and its token budget capped at our board's total. This is the real competitor, because tree decomposition is itself coverage assignment done inside one agent. The board beats it only if it achieves a higher AUROC or a lower score error at equal tokens, or a clearly smaller manipulation shift.
For calibrated scoring on ReviewArena-Eval, the strongest reported numbers are a mean absolute error of 0.700 to human ratings by Qwen3.6-Plus and an accept/reject score gap of 1.40 points for GLM-5.1, against roughly 0.5 for typical models (ReviewArena). We report our gap and our error against those figures, using the same scale and split. We do not claim a gain unless the gap exceeds 1.40 or the error falls below 0.700. Because our backbone may differ, we also rerun the best available of those models as a flat reviewer under our protocol.
For grounding, the reference is o1's claim-association F1 of 28.5, against 45.8 for human annotators on CLAIMCHECK.
The key ablation is a two-by-two: coverage assignment on or off, crossed with evidence checks on or off, all within the same hierarchy. This separates the two ingredients from the mere presence of an editor. A further ablation gives the associate editor the full paper. That tells us whether verification can substitute for the editor reading everything.
Why not just…?
Why not just give every reviewer the full paper with a long-context model? We include this as an upper reference at matched total tokens, which means fewer full readers. TreeReview reports that long-context reviewing still loses the middle of papers and drifts toward generic critique, but that is its authors' motivation rather than a controlled result. If full readers at equal cost match the assigned-coverage board, our hierarchy is unnecessary for LLMs, and we will say so. The human-relevance argument would then rest only on the claim that human readers cannot buy longer context.
Why not just use TreeReview, a single agent that decomposes the paper itself? This is the most serious threat. Decomposition inside one agent may decorrelate coverage just as well as assignment across agents. Where a board could still win is that one agent's errors on different leaves share the same priors and the same susceptibility to framing, while separate agents with a separate verifier do not. The matched-budget comparison, together with the manipulation tests, settles this. If TreeReview ties on accuracy and on robustness, the contribution reduces to evidence-carrying reports. That ingredient can be grafted onto TreeReview, and we would report it that way.
Why not just add more independent reviewers and average? The equation above predicts this stalls at $\rho\sigma^2$. AgentReview's finding that discussion cut rating variance by 27.2% through conformity suggests that interaction can raise correlation further, rather than lower it. Our sweep over $k$ measures the floor directly. If $\rho$ under skimming turns out to be no higher than under random-chunk reading, the core mechanism is wrong.
Why not just fine-tune a model on real review outcomes? FIRSTPASS reports 80.5% accuracy and 78.2% macro-F1 for a fine-tuned 7B model predicting the number of review rounds. That is a different task, and it uses the full review dialogue rather than reading the paper. ReviewArena notes that open models imitate review text while compressing scores. Fine-tuning is complementary: a fine-tuned reviewer could sit inside our board. We include one fine-tuned flat reviewer as an extra comparison if compute allows, but we do not treat it as a substitute for the structural question.
Proposed Experiments
The main experiment uses the 1,002-paper ReviewArena-Eval split. It crosses four reading policies with two report formats, inside the three-tier board and inside the flat baselines, at three total-token budgets. We report accept/reject AUROC, accept/reject score gap, mean absolute error to human scores, and estimated $\rho$.
The grounding experiment runs evidence-carrying and free-form boards on the 41 CLAIMCHECK papers. It measures claim-association F1 of the final meta-review's weaknesses against the human annotations. It also measures the fraction of weaknesses the verifier rejects that annotators also judge ungrounded, which tests whether the verifier filters the right things.
The robustness experiment applies the prestige-framing, cautious-language and evidence-free-rebuttal perturbations of Wang et al. We use their 100-paper set if available. Otherwise we recreate the protocol on a stratified subset of ReviewArena-Eval, and in that case we report shifts relative to our own unperturbed runs rather than comparing directly to their numbers.
Because ICLR and NeurIPS papers may appear in model training data, we repeat the main comparison on TMLR and COLM papers in ReviewArena and check that the ranking of configurations holds.
The decisive check is a single prediction. At equal total tokens, the board with assigned coverage and evidence checks should show lower reviewer-error correlation than the skimming board. It should show a higher AUROC than both independent averaging and budget-matched TreeReview, and a rebuttal-inflation shift well under the +0.53 reported for single referees. Each component of the two-by-two ablation should contribute.
If assigned coverage does not lower $\rho$, the correlated-error account is false. If the board loses to TreeReview at equal cost, editorial hierarchy is the wrong lever for LLM review and the useful contribution reduces to evidence-carrying reports.
References
- Accelerated Fragmentation: A Temporal Shift Analysis of Video Playback Behavior in the Context of New Media
- Why We Read Wikipedia
- Research on the Impact of Content Fragmentation Propagation in Short Videos
- The Impact of Short Videos on College Students' Cognitive Abilities and Learning Effect: Roles and Mechanisms
- IEEE Transactions on Information Theory publication information
- Dilemmas and Governance Strategies in the Implementation of the Modern Apprenticeship System in Chinese Vocational Colleges: A Perspective from Principal-Agent Theory
- AgentReview: Exploring Peer Review Dynamics with LLM Agents
- TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review
- CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?
- When AI reviews science: Can we trust the referee?
- ReviewArena: A Large-Scale Cross-Conference Dataset and Benchmark for LLM Peer Review
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes