Headcount-Blind Deliberation: Letting the Area Chair Hide Who Agrees With Whom

When several language-model reviewers discuss a paper, the reviewers who are right often lose. The failure is not that they cannot see good arguments. It is that they can count heads. Each reviewer sees which peer said what, how many peers share a stance, and who changed their mind last round, so the discussion drifts toward the majority. Simulations of conference review find that discussion shrinks rating variance by 27.2% and that one biased reviewer flips the final decision in 37.1% of cases (AgentReview). Debate studies find that agents holding a correct answer abandon it under peer pressure at rates near 81% when there is no judge (Peacemaker or Troublemaker).

We propose borrowing a technique from network privacy, where the problem of "observers link your actions over time" was solved long ago. Privacy systems give a user fresh throwaway names at each stage of a workflow, and a trusted middle server holds the only map from those names back to the real identity. Peer review already has a trusted middle tier: the area chair. Our proposal makes the area chair tier act as that middle server for the discussion itself. Reviewers never see their peers' scores, how many peers hold each position, or which argument in round two came from the same author as an argument in round one. They see a deduplicated set of arguments under fresh pseudonyms each round. The area chair keeps the full linkage map, so accountability and final decisions are unchanged. In two sentences: conformity in multi-agent review comes from linkability (the ability to count and track stances), not from names alone. A hierarchy whose middle tier holds the linkage and exposes only arguments should keep correct minority objections alive better than blinding, independent drafting, or a central judge.

Core Claim

The testable claim is specific. Take a review committee with a fixed reviewer model, round budget and token budget. Route discussion through an area-chair proxy that removes three things: scores, stance counts (through merging duplicate arguments), and cross-round linkage (through rotating pseudonyms). This will raise the rate at which a correct objection held by a single reviewer survives into the final decision. It will do so without lowering accuracy when the majority is right. We also claim that most of the gain comes from removing counts and cross-round linkage, not from removing names. Plain name removal is therefore the baseline we most need to beat, not a component we get credit for.

Where This Sits in Prior Work

The identity-masking literature supplies the method. Session-level ephemeral identifiers (Li et al.) stop observers from linking separate interactions. The phase-based pseudonym scheme of Liu et al. uses different pseudonyms for different stages of one workflow, so even a curious coordinator cannot join the stages. We reuse that stage-splitting idea, but the "observer" we defend against is the peer reviewer rather than a server. Wang et al. show that hashing a fixed identifier just creates a new trackable name. That is the motivating analogy for why static labels such as "Reviewer 2" do not stop tracking across rounds.

The edge-server identity-replacement system (Edge Computing and Identity Replacement) supplies our architecture. An intermediate server swaps identifiers on outgoing requests instead of perturbing the content. It reports under 10% overhead and near-zero "privacy injury" on a digital-library query log. Our area-chair proxy plays the same role, except that it rewrites deliberation messages rather than database queries.

The automotive benchmark of pseudonym signatures (CAV comparative evaluation) constrains the design. There, pairing-based identity signatures took 946 ms to verify on ARM and group signatures inflated messages by up to 127%. Because our threat is social inference by peers, not a network adversary, we deliberately use no cryptography. The trust already present in the hierarchy does the work.

The peer-review and debate literature motivates the proposal and supplies the evaluation. AgentReview runs a five-phase simulation of conference review with persistent reviewer identities throughout, and measures conformity and area-chair style effects (30.6% of decisions). We keep its pipeline and change only what reviewers can see during the reviewer–area-chair discussion phase. Agent Reviewers is a hierarchical multi-agent reviewer with shared memory. It is the strongest decision-prediction system in the file, and its discussion tier is fully transparent. Voting or Consensus? introduces All-Agents Drafting, in which each agent writes independently before seeing peers, and reports a 3.3% average gain. Independence only at the start is exactly the alternative we must beat. Peacemaker or Troublemaker defines disagreement collapse rate, the fraction of correct dissenters who fold. It shows that a central judge cuts this rate from about 81% to 41% on CommonsenseQA and that debate benefit peaks at 2–3 rounds. We adopt its metrics and its round budget.

One paper constrains how we read results. Articulate Intuition or Genuine Analysis? finds that LLM meta-reviewers reward fluent form, that length explains about 74% of an apparent AI-review quality edge, and that acceptance tier does not predict reasoning quality (p = 0.260). This cuts against using real accept/reject labels as the only ground truth, and it is why our decisive test uses planted flaws with known answers.

Two further papers motivate the linkage-holding tier but are out of scope. Identity Theft in AI Conference Peer Review documents 94 fraudulent reviewer profiles. Calibration with Privacy in Peer Review shows that public score calibration leaks reviewer assignments. Both suggest that keeping linkage in one accountable tier has value beyond deliberation quality, but we do not test fraud or calibration here.

Research Program

We build the proxy as a thin layer inside the AgentReview discussion phase and inside the Agent Reviewers discussion tier. After independent drafting, each reviewer's review is split by an extraction model into atomic claims, each tagged as a strength or a weakness. Scores are withheld.

The proxy then does three things. First, it clusters semantically equivalent claims and emits one representative per cluster, with no count attached. This hides the headcount. Second, it assigns each representative a fresh random pseudonym drawn anew every round. Reviewers can refer to "argument K7" within a round but cannot link K7 to last round's arguments or to a peer's score. Third, as a separable option, it paraphrases each representative into a fixed neutral register. This removes stylistic fingerprints and the length and fluency cues shown to sway LLM judges.

Reviewers revise their claims and private scores for at most three rounds. The area chair sees everything linked, including who holds what, scores, and flips. It writes the meta-review and decision as in the unmodified pipeline. We also run a variant in which the area chair sees only the masked view, to test whether masking the upper tier helps or whether hierarchy is useful precisely because the chair keeps linkage.

The analysis asks where conformity lives. We measure, per round, how strongly a reviewer's stance change is predicted by the visible majority versus by the content of the arguments they are shown. We expect the majority term to fall sharply under counting and linkage removal and only mildly under name removal.

Baselines

The naive reference point is the unmodified AgentReview discussion phase. Reviewers there keep static labels and see peers' reviews and scores. Its reported effects are the 27.2% variance drop and the 37.1% flip rate from a single biased reviewer, and it tells us how much conformity there is to remove. The proposal beats it if dissent survival rises with decision accuracy held equal or better.

The strongest existing system for decision prediction is Agent Reviewers. On ICLR 2024 it reports 0.70 accuracy and 0.605 F1 for accept/reject prediction, against 0.61 accuracy for a single agent, at $0.65 per 100 papers with GPT-4o-mini. We rerun it on the same papers with GPT-4o-mini, then insert our proxy into its discussion tier with everything else fixed: memory, reviewer selection, model and token budget. The claim is beaten only if the masked version matches or exceeds 0.70 accuracy on real decisions and also improves planted-flaw survival. A gain on planted flaws alone, with a loss on real decisions, would count as a trade-off, not a win.

The strongest method aimed at conformity is the central judge from Peacemaker or Troublemaker. It reports a disagreement collapse rate of roughly 41% on CommonsenseQA, against about 81% for decentralized debate. We rerun it with Qwen3-32B and Llama 3.3-70B at 2–3 rounds and the same token budget. The proposal must push collapse clearly below the judge's rate on the same model, without raising the negative agreement rate, i.e. how often agents converge on a wrong answer.

The independence-only baseline is All-Agents Drafting from Voting or Consensus?: independent first drafts, then open deliberation. Our method includes the same drafting step, so any gain over it is attributable to masking after the draft.

The ablations isolate the key ingredient. We run name removal only, where claims are shown unlabeled but not merged and scores stay hidden. We add counting removal, then cross-round rotation, then paraphrase normalization, one at a time. Finally we run the full proxy with the area chair also masked. All conditions share the same model, seeds, rounds, token caps and prompt-tuning budget: a fixed small number of prompt revisions per condition, chosen on a held-out split. If name removal alone captures most of the gain, the central claim is wrong, and we will report it that way.

Why not just…?

Why not just use standard double-blind labels like "Reviewer 1"? Static labels are exactly the persistent pseudonym that Wang et al. show remains trackable. AgentReview's 27.2% variance collapse occurs with such labels in place. The naive baseline directly measures whether they are enough.

Why not just remove all speaker labels? This is the real threat, and it is the name-removal ablation. Unlabeled reviews still reveal the headcount: three texts saying "novel" and one saying "flawed" show the majority at a glance. We predict it helps less than removing counts. If it matches the full proxy, the simpler intervention wins and we say so.

Why not just draft independently first? All-Agents Drafting gives 3.3% (Voting or Consensus?), but it protects only the first round. The conformity AgentReview measures happens afterwards. Our design contains AAD as its first step, so the comparison measures exactly what post-draft masking adds.

Why not just give reviewers an independent, low-agreeableness persona? Prompting agents toward the "troublemaker" end of the persona scale in Peacemaker or Troublemaker is cheap and does reduce folding. The risk is that stubborn agents also reject correct corrections. We run the most independent persona as an extra baseline and report both dissent survival and the rate at which wrong minorities are correctly overturned. Masking should improve the first without hurting the second. A persona prompt should trade one for the other.

Why not just use a central judge? The hierarchy already has one, the area chair, and the judge baseline shows it roughly halves collapse. But the collapse it leaves happens among reviewers before the judge reads anything. The masked-versus-unmasked area-chair variant tests whether the two effects add up.

Proposed Experiments

Experiments run on three benchmarks. The first is the AgentReview ICLR corpus of 523 submissions from 2020–2023. The second is ICLR 2024 papers from Reviews-STD, the corpus used by Agent Reviewers, for real accept/reject labels. The third is the review-writing tasks of ResearchBench (200 tasks), to check that masking does not degrade meta-review similarity to human text. ResearchTown reports 0.49 semantic similarity on this set, and we treat that as a floor to stay near, not a target.

The decisive test uses planted flaws. On 200 papers we insert a verifiable fatal error, such as a table whose numbers contradict the stated claim or a proof step with a missing assumption. We give only one of three reviewers the relevant section in full, creating a correct minority by construction. The main metric is the fraction of papers where that objection appears in the meta-review and flips the decision to reject.

The mirror test covers the case where the majority is right. We plant a spurious but fluent objection in one reviewer's draft and measure how often it is wrongly adopted. We further check robustness by cross-scoring meta-reviews with the Reasoning Quality Score from Kahneman4Review, with length controlled.

For a clean comparison outside peer review, we repeat the masked-versus-judge contrast on CommonsenseQA and MMLU-Pro. We report disagreement collapse rate, negative agreement rate and sycophancy score, so results are comparable to the ~81%/41% collapse figures.

The idea succeeds if three things hold. Planted-flaw survival rises substantially over both AAD and the central judge. Spurious-objection adoption does not rise. Real-decision accuracy stays at or above Agent Reviewers' 0.70. It fails, informatively, if name removal alone matches the full proxy, or if gains appear only when paraphrase normalization is on. That second outcome would mean the effect is about style rather than linkage.