Why LLM Reviewers Hedge: Testing Editorial Hierarchy Through Critique Grounding

Language-model reviewers score papers in a narrow middle band. On a 10-point scale they separate accepted from rejected papers by only about half a point, even when given the venue's own instructions and scale. This proposal asks whether that compression comes from the reviewer's institutional position rather than its raw ability. A human reviewer who knows an area chair will check every criticism tends to commit to specific, verifiable weaknesses. A reviewer writing into a void hedges. We propose to place LLM reviewers inside explicit editorial hierarchies and measure which intermediate quantities carry the effect of that structure onto scores. The hierarchies are reviewer, area-chair auditor, and editor-in-chief policy layer. The candidate intermediate quantities are how many grounded criticisms the reviewer makes and how strongly it commits to them.

The method comes from an unlikely place: the "mediation" literature in environmental and organizational psychology. This literature models how an external condition, such as a workplace climate or a school's exercise culture, changes behavior only by first changing an internal state, such as perceived support or self-control. Those studies are almost all cross-sectional surveys, so their causal arrows remain contested. With LLM reviewers we can do what survey researchers cannot. We can randomize the editorial structure, and then directly overwrite the proposed intermediate state to see whether the effect follows it. The result would be a causal decomposition of score compression into "structure acts through critique grounding" and "structure acts some other way." It would also give a practical answer to whether hierarchy is worth its cost over cheaper fixes.

Core Claim

We claim that hierarchical editorial oversight improves an LLM reviewer's ability to rank accepted above rejected papers, not merely to stretch its score range. We further claim that this improvement runs through a serial chain. First, oversight increases the number of weakness claims that an independent checker can verify against the paper text. Second, more verified weaknesses raise the reviewer's committed confidence. Third, higher confidence produces more decisive scores.

The chain is falsifiable in two ways. If oversight improves ranking but transplanting the verified critiques into an un-overseen reviewer does not, then the effect does not travel through grounding. If oversight only widens the score spread without changing ranking, then it is no better than post-hoc rescaling.

The distinction between ranking and spread is the key design choice. The headline ReviewArena metrics, mean absolute error and the mean accept/reject gap, can both be improved by a monotone rescaling of scores, which requires no structure at all. Area under the ROC curve (AUC) for accept versus reject cannot be changed by any monotone transform. Ranking is therefore where hierarchy has to prove itself.

Grounding in Prior Work

The editorial side supplies the structures we instantiate. IEEE and VLDB governance documents describe editorial authority split into tiers. An executive sets universal policy and associate editors run domain-specific filtering (IEEE Transactions on Information Theory publication information, IEEE Transactions on Cybernetics publication information, Front Matter). These documents describe roles, not effects; we turn the roles into manipulable conditions.

Principal-agent analyses of vertical chains motivate both the expected benefit and the expected cost. The analyses cover government delegating to colleges delegating to firms (Dilemmas and Governance Strategies in the Implementation of the Modern Apprenticeship System…) and bank headquarters delegating to branches (Credit Risk Management of Commercial Banks in China). Unmonitored agents drift toward local convenience, but long chains dilute information.

Policy-as-Prompt constrains our design. It found that semantically identical moderation policies, formatted differently, reach similar average accuracy (about 0.79) but disagree on edge cases, and that verbose rule-stuffed prompts do worst. Any effect we attribute to "hierarchy" must therefore survive paraphrase of the structural prompts, or it is just prompt multiplicity.

Two papers show that hierarchy helps LLM teams, but neither involves an evaluative task or explains why. Embodied LLM Agents Learn to Cooperate in Organized Teams reports that designated leaders cut household-task completion time, for example by 9.76% for three GPT-3.5-turbo agents. Mindstorms in Natural Language-Based Societies of Mind finds a monarchical society reaching 67.42% on visual question answering. We move the question to peer review and ask for a mechanism, not just an outcome.

The closest existing systems leave the mechanism open. ResearchTown simulates review as flat graph message passing, and its review writing reaches only 0.49 semantic similarity to human reviews, against 0.68 for paper writing. ReviewArena benchmarks reviewers in isolation. Neither places the reviewer under oversight, and neither asks what internal quantity carries institutional pressure into the score.

The psychology side supplies the analytic template. Serial chain mediation, in which leadership raises perceived support, which raises commitment, which drives behavior, is the form tested by Liu, Jia and Su and Chen et al.. Ye and Lu show the case we must allow for: the upstream lever has no direct effect once the chain is accounted for. Siebers and Aickelin flag the field's central weakness: the causal direction in observational path models is unresolved. Interventional mediator manipulation is our answer to that weakness. We borrow the model form, not the survey method.

Research Program

The treatment $T$ is the editorial structure in which a reviewer operates. We test four structures, all on the same papers:

  • Solo reviewer. The reviewer receives the venue instructions, matching ReviewArena's protocol.
  • Flat panel. Three reviewers score independently and their scores are averaged.
  • Two-tier hierarchy. Reviewers are told an area-chair agent will audit each weakness claim against the paper. The auditor actually does so, returning unsupported claims for revision before the final score.
  • Three-tier hierarchy. An editor-in-chief agent first condenses venue policy into guidance passed down through the area chair. This tests information dilution along longer chains.

The mediators are measured from the review text. The first, $M_1$, is the count of weakness claims that a separate verifier model, given the paper, confirms are present and accurately stated. The second, $M_2$, is committed confidence: the reviewer's stated confidence combined with a hedging-language score. The outcome $Y$ is the numeric score; at the paper-set level it is summarized by decision AUC.

We use the standard decomposition. The total effect of moving from the solo reviewer $t_0$ to a hierarchy $t_1$ splits into a natural indirect effect through the mediators and a direct remainder:

$$ \mathrm{TE} = \underbrace{E[Y(t_1, M(t_1))] - E[Y(t_1, M(t_0))]}_{\text{indirect, via } M} + \underbrace{E[Y(t_1, M(t_0))] - E[Y(t_0, M(t_0))]}_{\text{direct}} $$

Here $Y(t, m)$ is the score a reviewer gives under structure $t$ when its critique set is fixed to $m$, and $M(t)$ is the critique set that structure $t$ naturally produces.

In survey mediation studies the cross-world term $Y(t_1, M(t_0))$ is unobservable. With LLMs we can construct it directly. We take the verified critique list a reviewer produced under one structure and transplant it into a reviewer under the other, then ask that reviewer for a score. This "mediator transplant" is the core technical move.

We also run a reverse transplant: we hand a solo reviewer a critique list from a different paper with matched length. This checks that it is the paper-specific grounding, not critique volume, that moves scores.

Baselines

Solo reviewer (naive reference). This is the naive reference: ReviewArena's protocol of a single reviewer with venue instructions. The strongest reported numbers for this protocol are the best mean absolute error of 0.700 (Qwen3.6-Plus) and the largest accept/reject gap of 1.40 points (GLM-5.1), against an average gap of about 0.5 (ReviewArena). There is no reported result on Lacuna for decision AUC on ReviewArena-Eval, so we will compute it ourselves for every model under this protocol, on the full 1,002-paper subset.

Post-hoc monotone recalibration (strongest cheap competitor). We fit isotonic regression from solo-reviewer scores to human scores, separately per venue, on held-out folds. This will likely match or beat our hierarchy on mean absolute error and gap. That is exactly why AUC is the decisive metric: recalibration leaves AUC unchanged by construction.

Pairwise comparison ranking (strongest structure-free competitor). The reviewer compares two papers from the same venue at a time, and a Bradley-Terry fit turns the comparisons into a ranking. This attacks ranking directly without any hierarchy.

Flat self-consistency panel. This is the averaged three-reviewer panel. It controls for the extra compute the hierarchy uses.

Fair comparison. All conditions are run with the same backbone models and the same papers. Each condition is capped at the same total token budget as the two-tier hierarchy; pairwise comparison and self-consistency get as many calls as fit within that cap. Prompts are tuned on a shared development split disjoint from evaluation.

Placebo-audit ablation. This isolates the key ingredient. Reviewers are told an area chair will audit their claims, but the auditor returns nothing. If the placebo matches the real audit, the effect is accountability framing, not verification.

Beating the baselines. The hierarchy must beat recalibration and self-consistency on AUC. It must also match or exceed pairwise ranking on AUC at equal or lower cost. Both conditions must hold across at least three paraphrases of every structural prompt.

Why not just…?

Why not just rescale the scores after the fact? Rescaling fixes spread, not discrimination. If compression were a pure range problem, isotonic recalibration would solve everything, and our proposal would add nothing on mean absolute error. We concede that point in advance and stake the claim on AUC, which no monotone transform can change. If the hierarchy's AUC gain is zero, the honest conclusion is that compression is cosmetic and rescaling is the right fix.

Why not just ask for pairwise comparisons? This is the most serious threat. Comparative judgment may avoid hedging entirely, because there is no absolute scale to retreat to the middle of. We run it at matched budget. If pairwise ranking reaches the hierarchy's AUC, we will report that structure is unnecessary for ranking. The mediation analysis would then still say whether grounding explains the residual ranking errors of both methods.

Why not just average more independent reviewers? Averaging reduces noise in a reviewer's judgment but not bias in it. If every reviewer hedges for the same reason, averaging keeps the hedge. Organized Teams reports that unorganized LLM teams produce redundant, over-compliant exchanges, which suggests flat panels converge rather than sharpen. The matched-compute self-consistency baseline tests this directly on review.

Why not just write a stricter rubric? Policy-as-Prompt found that verbose rule-heavy prompts performed worst and that equivalent rules disagree on edge cases. ReviewArena already supplies full venue instructions and still observes compression. We include a detailed-rubric variant of the solo reviewer for completeness.

Proposed Experiments and Decisive Checks

Main experiment. We run the four structures, the baselines, and the transplant conditions on ReviewArena-Eval, using several open-weight models from the ReviewArena evaluation. These include Gemma-4-26B, which ReviewArena reports as "consistently inflated," as a stress case. We report three things: AUC per venue; the per-venue total effect split into direct and indirect parts; and the indirect effect split further into the path through $M_1$ alone and the serial path $M_1 \to M_2$.

First decisive check: does the chain carry the effect? Transplanting two-tier verified critiques into solo reviewers should recover most of the hierarchy's AUC gain. Mismatched-paper critiques of the same length should recover none of it. If the matched transplant recovers little, the chain claim fails even if the hierarchy helps.

Second decisive check: the cost of tiers. Moving from two to three tiers is the principal-agent test. If the editor-in-chief's condensed policy lowers $M_1$ and AUC relative to two tiers, we have measured information dilution along an editorial chain in a controlled setting.

Third check: robustness to prompt wording. Every effect is re-estimated across prompt paraphrases. Effects whose sign flips under paraphrase are reported as multiplicity, not structure.

Secondary observational analysis. On the 196,099 human reviews in the full ReviewArena corpus, we examine whether reviews with more specific weakness claims, as counted by the same verifier, carry more extreme scores. We also check whether this association differs between TMLR's action-editor model and the conference area-chair model. This analysis is explicitly correlational and subject to the causality caveat raised by Siebers and Aickelin. Its role is only to ask whether the mediator we identify in LLMs plausibly matters for humans too.