Shared Anchor Papers: Calibrating Area Chairs, Not Just Reviewers

When a conference accepts a paper, we would like to say the paper was good. The decision also depends on who handled it. In LLM-agent simulations of ICLR review, simply switching the area chair (the mid-level editor who reads the reviews and makes the call) from an "authoritarian" style to a "conformist" style changed 30.6% of final decisions (AgentReview). Education research calls this the attribution problem: a teacher's style can swamp the effect of the curriculum being tested (Learning Mathematics from Examples and by Doing), and course-performance models break when they ignore which instructor taught each section (Examining students' course trajectories). Editorial boards have the same confound one level up. Methods that correct reviewer bias stop at the reviewers and leave the editor's own tendencies untouched.

We propose to redesign the senior tier of an editorial hierarchy (senior area chairs or program chairs) so that its job is to measure and remove each area chair's personal tendencies rather than to re-judge papers. The mechanism is a small set of shared "anchor" papers. Each anchor is handled independently by several area chairs. The overlap between area chairs on these anchors lets the senior tier estimate, for each area chair, how lenient they are, how sharply they separate strong from weak work, and how much they defer to reviewers versus their own reading. It then adjusts that area chair's decisions on the rest of their batch. The central claim is that a few well-placed anchors per area chair remove most style-driven decision flips, and do so at a fraction of the cost of the obvious fix, which is to give every paper two area chairs.

Core Claim and What Is New

Least Square Calibration for Peer Review supplies the key technical idea. It treats each reviewer's score as a personal function of latent paper quality and proves that, without noise, both a reviewer's offset and slope are recoverable only if the reviewer–paper graph is "doubly connected," meaning reviewers share at least two papers. That work models one flat layer of numeric scores. It never asks how an editor tier uses or distorts those scores, and it cannot see editors at all.

We move this identifiability argument up one tier and extend it to the behavior that actually separates area-chair styles. Let $m_{jp}$ be the meta-score area chair $j$ gives paper $p$, $q_p$ the paper's latent quality, and $\bar r_p$ the reviewer-calibrated mean score. We model

$$m_{jp} = \alpha_j + \beta_j q_p + \gamma_j(\bar r_p - q_p) + \varepsilon_{jp},$$

where $\alpha_j$ is leniency, $\beta_j$ is discrimination, $\gamma_j$ is deference to reviewers (a conformist has $\gamma_j$ near 1, an authoritarian near 0), and $\varepsilon_{jp}$ is noise. Separating three parameters per area chair needs overlap beyond double connectivity. We conjecture that each area chair needs at least three anchors, connected through other area chairs, on which reviewers and latent quality disagree by varying amounts. We will test this conjecture empirically and do not claim to prove it.

Three things are new. The calibration target is the editor rather than the reviewer. The senior tier's role is redefined as designing the overlap graph and fitting this model, rather than performing a second subjective read. And the editors' traits are hidden from the calibrator. That last point matters because textual traces of individual differences are hard to read directly: human judges reached only Fleiss' kappa 0.53 when classifying simulated students' ability from dialogue (Foundations of PEERS). The calibrator must therefore infer style from the overlap, not from the prose.

Research Program

We will extend the AgentReview five-phase simulation with a third tier, run on its 523 ICLR 2020–2023 submissions. Area chairs will be uncontrolled in two ways at once. First, their personas will be sampled from AgentReview's authoritarian, conformist and inclusive prompts. Second, they will run on different LLM backbones, so that they also differ in ways no prompt specifies. This second source of variation is deliberate. In the real world nobody controls an editor's personality, and a method that only works when styles are known would be uninteresting.

Each area chair will output a meta-score alongside a decision. Below the area-chair tier, reviewer scores are first calibrated with LSC so that our gains are attributable to the editor tier alone. The senior tier then does four things in turn: it chooses which papers become anchors, collects independent anchor meta-scores, fits the equation above across the anchor graph, and re-ranks all papers by their estimated $q_p$ against a fixed acceptance rate.

The analysis has four parts. First, how the anchor count $k$ per area chair trades off against residual style sensitivity. Second, whether a graph designed for connectivity beats randomly chosen anchors. Third, whether $\gamma_j$ is recovered better than chance against each agent's hidden persona. Fourth, whether the extra anchor workload provokes the "altruism fatigue" AgentReview observed, in which one irresponsible reviewer led peers to cut their discussion length by 18.7%.

Baselines

The naive reference point is the AgentReview default: one area chair per paper, with reviewer scores calibrated by LSC, deciding on its own. This baseline isolates how much harm remains once reviewer-level calibration is done.

The strongest existing calibration method is LSC itself, applied flatly. Here area chairs are treated as additional "reviewers" in one bipartite graph, without the tiered structure or the deference term. LSC is the right competitor because it is the only method in the file with a proof of identifiability. Its reported numbers are 80.1% precision at identifying the top 10% of papers on synthetic data with noise σ = 0.5, against 39.2% for averaging, and 82.2% precision on a real CS peer-grading set (Least Square Calibration for Peer Review). It receives the same anchor overlap as our method, so any gain comes from modeling deference and the hierarchy rather than from extra data.

The redundancy baseline gives every paper two independent area chairs, with a senior-chair agent breaking ties. This is the practical incumbent and the most serious threat. It is compared at matched area-chair workload, measured as the total number of area-chair–paper handlings, and also at its natural cost of double coverage.

Two ablations isolate the key ingredient. One fits the model with $\gamma_j$ fixed at zero, which reduces to a leniency-and-slope correction. The other replaces model fitting with per-area-chair z-scoring of meta-scores, using no anchors. Every condition uses the same papers, backbones, persona draws, random seeds and prompt-tuning budget.

Beating the baselines means the following. Our method must lower the style-flip rate below both the redundancy baseline and flat LSC at no more than about 1.3 times the single-area-chair workload, while redundancy costs 2 times. It must do this without lowering agreement with the real ICLR decisions.

Why not just…?

Why not just give every paper two area chairs? Averaging two draws reduces style variance but does not remove a shared bias. If both backbones lean lenient, the error stays. Redundancy also doubles the workload on the tier where AgentReview already saw engagement collapse. It is a real threat, and it could win when area-chair styles are independent and roughly symmetric. The matched-workload comparison will settle this. If redundancy matches us at equal cost, the anchor idea is not worth its complexity and we will say so.

Why not just instruct every area chair to be "inclusive"? In a simulation this homogenizes styles by fiat, and real editors cannot be reprogrammed this way. Our backbone-heterogeneity condition tests the point directly. We run all area chairs with the identical inclusive prompt and measure the flip rate that remains between backbones. If it is near zero, prompting suffices in silico, and the case for our method rests on real-world editors.

Why not just skip area chairs and rank papers by calibrated reviewer scores or pairwise tournaments? Removing the editor removes the editor confound, but it also removes the editor's job of weighing arguments. Pairwise tournaments flatten the hierarchy and amplified institutional prestige bias, reaching a higher Gini coefficient than human committees (From Replication to Redesign). Even so, flat LSC ranking without any area chair will be run as an extra condition. If it agrees with ICLR decisions as well as calibrated area chairs do, the editor tier is adding noise rather than judgment, which would itself be a useful finding.

Proposed Experiments and Decisive Checks

The primary metric is the style-flip rate: the fraction of decisions that change when the area-chair assignment is re-randomized across personas and backbones. We first reproduce AgentReview's 30.6% figure before applying any fix.

Secondary metrics are agreement with actual ICLR decisions on the 523 papers and top-10% precision against those decisions. We also compute the correlation between recovered $\gamma_j$ and the hidden persona, and the meta-review reasoning quality using the Reasoning Quality Score rubric from Kahneman4Review. Because that work found review length explained about 74% of a naive quality gap, we will report reasoning quality with length controlled.

As an external sanity check, we will use ReviewArena, which reports that LLM scores separate accepted from rejected papers by only about 0.5 points. We will confirm that our area-chair agents' meta-scores separate real ICLR accepts from rejects at least that well, so we are not calibrating a decision rule that is already flat.

There are three decisive checks. The style-flip rate must fall sharply as $k$ rises from 0 to 3 and then plateau near the level set by noise, which is what the identifiability argument predicts. Designed anchors must beat random anchors at equal $k$. And the $\gamma_j$ ablation must lose specifically on papers where reviewers and the area chair disagree.

We also include a negative control. Prestige halo effects, which shifted 27.7% of decisions in AgentReview, are tied to individual papers rather than to editors. We therefore predict anchors will not reduce them, and we will report that result as a boundary on what this design can fix.

The main limitation is that real area-chair identities are not public in the datasets in the file. Validation on real data is therefore limited to the decision-level agreement and the score-separation check described above.