Conference peer review has a built-in conflict of interest: many reviewers are also authors, and a reviewer can help their own paper by quietly scoring competitors low. One known fix separates the two roles. Authors are split into groups that share no co-authorship ties, and each group only reviews papers from other groups, so a reviewer's scores can never reach their own paper's ranking. The cost is that no one ever compares the groups with each other. The system has no principled way to decide how many acceptances each group should get, or whether a "6" from one group means the same as a "6" from another. If one group happens to hold stronger papers, fixed per-group quotas quietly reject good work and accept weaker work.
We propose adding a third tier, borrowed from how editorial boards divide their work. Between the chief editor, who fixes the total number of acceptances, and the cross-group reviewers, who rank papers within each group, sits a disinterested calibration tier. This tier has no submissions, never sees reviewer identities or reports, and only decides each group's acceptance quota. The natural candidate is a language-model reviewer. Its per-paper judgments are known to be poor, but averaged over hundreds of papers they may be good enough to estimate relative group quality. Because quotas then depend only on paper texts and on a party with no stake, strategyproofness is preserved exactly. Our claim is narrow and testable: an unreliable, stakeless evaluator is useless for ranking papers but useful for splitting a quota across groups, and this recovers much of the accuracy that conflict partitioning gives up.
Core Claim
On Strategyproof Conference Peer Review introduces Divide-and-Rank. It partitions the reviewer–author conflict graph into disconnected groups, has each group review only other groups' papers, and merges the sub-rankings with a topological procedure called Contract-and-Sort. It proves that full pairwise unanimity cannot coexist with strategyproofness. It also shows on ICLR 2017 data that removing about 7% of prolific authors shrinks the largest conflict component from 133 papers to 8, so partitioning is practical. The paper itself flags what is left open: groups are never cross-calibrated. As we read the construction, any strategyproof version must set the between-group split without using the reports of reviewers who are authors in those groups. That leaves a narrow opening. The split can use any signal that is independent of all reports.
We fill that opening with a separated duty. Let $k$ be the total number of acceptances, $n_g$ the number of papers in group $g$, $N=\sum_g n_g$, and $s_g$ the share of group-$g$ papers in the calibration tier's global top-$k$. The quota is
$$k_g = \mathrm{round}\Big(k\,\big[\lambda\, s_g + (1-\lambda)\, n_g/N\big]\Big),$$
where $\lambda\in[0,1]$ controls how far the quota may move from plain size-proportional allocation. We set $\lambda$ from the calibration tier's measured reliability on held-out venues, not from the current conference's reports. Within each group, the top $k_g$ papers are chosen by the other groups' reviewers exactly as in Divide-and-Rank.
Strategyproofness holds for a simple reason. A reviewer's report affects only rankings inside groups they are not in, and never affects any $k_g$. The new question is empirical: is $s_g$ informative enough that $\lambda>0$ helps?
The case for optimism is statistical. ReviewArena finds that most language models separate accepted from rejected papers by only about 0.5 points on a 10-point scale, which is useless per paper. But a group mean averages this signal over $n_g$ papers. Independent per-paper noise then shrinks roughly as $1/\sqrt{n_g}$, by the usual standard-error argument. What does not shrink is systematic bias, for instance a model favouring certain topics. Groups defined by co-authorship will correlate with topic. Whether topic bias swamps the averaged quality signal is the question on which the proposal stands or falls.
Relation to Prior Work
Two mechanism papers motivate and constrain the design.
Impartial Peer Review gives the CREDIBLE SUBSET rule for the setting where each of $n$ author-reviewers scores $m$ proposals and at most $k$ winners are picked. The rule is exactly impartial and achieves a tight $k/(k+m)$ approximation, but it can return fewer than $k$ winners. It shows that exact impartiality has a real accuracy price. Our mechanism always fills exactly $k$ slots, and it pays its price in a different place: it relies on an outside signal rather than on randomization.
Divide-and-Rank supplies the within-group machinery we reuse unchanged. We alter only the one component its authors left unspecified.
The separation-of-duties literature supplies the design principle rather than a technique. Split-entity systems such as the group-signature toll-pricing architecture (A Group Signature Based Electronic Toll Pricing System) and the social-server/location-server split in PPLS keep "who" and "what" in parties that cannot see each other's data. We apply the same idea to decision rights. The party that decides how much each group gets must not see who reviewed what. The party that decides which papers win must not influence its own group's slot count.
BEMPAS similarly splits identity, behavior, and payoff across three ledgers for employee evaluation. It handles only numeric sensor data and relies on a tamper-proof ingestion layer. Our calibration tier faces the analogous threat: authors may try to manipulate the language model through the paper text. We address this explicitly below.
The editorial-hierarchy work motivates the tiering. The standard IEEE board model (IEEE Transactions on Cybernetics publication information) separates policy-setting, mid-tier area handling, and operational reviewing. ARGO (An Adaptive Responsible AI Governance Framework for Decentralized Organizations) describes a shared-foundation tier that absorbs failures of local units. Both are qualitative. We give one tier a formally specified, report-independent job.
Embodied LLM Agents Learn to Cooperate in Organized Teams shows that structured roles help language-model teams. Its setting is cooperative, not adversarial, and we do not rely on it beyond motivation. ResearchTown simulates review with no conflicts of interest, so it does not bear on strategyproofness.
Research Program
We will build a semi-synthetic conference simulator on the ReviewArena corpus. The corpus contains 51,529 papers and 196,099 reviews across seven venues, with scores and final decisions. For each venue-year, the simulator first builds a conflict graph. It uses author metadata where the corpus provides it. Where it does not, it imposes the component-size distribution reported for ICLR 2017 (over 250 components; largest component of 8 papers after pruning). The simulator then partitions papers into groups, following the Divide-and-Rank procedure.
The critical manipulation is controlled between-group quality skew. We assign papers to groups so that group acceptance rates differ by chosen amounts, from equal to twofold. We also add per-group reviewer scale shifts, such as one group scoring everything half a point lower, which mimics the calibration failure that partitioning leaves unaddressed. Real human scores play the role of cross-group reviews. Final decisions and mean human scores serve as the quality reference.
The calibration tier is an open-weight language model. We use the same model family and prompts ReviewArena benchmarks, including venue-aware prompting, and it scores every paper once. It receives only the paper text; it never sees reviews or identities. We will measure how the error of $s_g$ scales with group size, and we will separate random noise from topic-driven bias by comparing raw group means against means computed within subject areas. If topic bias dominates, the program pivots to area-stratified quotas: $s_g$ is computed within each area and then summed, which removes between-area differences the model has no business judging.
We will also test adversarial robustness. Authors in one group insert hidden instructions or inflated self-assessments aimed at the model. We measure how far this moves $k_g$, and we show that $\lambda$ caps the worst-case shift, since a manipulated group can gain at most $\lambda k$ slots beyond its proportional share.
Baselines
The naive reference point is Divide-and-Rank with size-proportional quotas, which is our mechanism with $\lambda=0`$. It is strategyproof and is the natural default when groups cannot be compared. It is the baseline the proposal must beat: the claim is exactly that $\lambda>0$ improves accuracy at no strategyproofness cost. We run it on identical simulated conferences, with identical partitions and within-group rankings, so any difference comes from the quotas alone. No reported result was found on Lacuna for Divide-and-Rank's selection accuracy on real review data, so we establish its accuracy ourselves.
The strongest non-strategyproof competitor is the standard practice: rank all papers globally by mean reviewer score and take the top $k$, with only individual conflict checks. We run it twice. With honest scores it is an accuracy ceiling. With a fraction of reviewers adopting the competitor-lowering strategy described in Impartial Peer Review, it shows what that accuracy costs. The mechanism beats this baseline if, at realistic manipulation rates, its overlap with the reference accept set exceeds the manipulated global ranking's overlap. Separately, our strategic reviewers' own-paper gain must stay at zero, which holds by construction and which we verify empirically.
CREDIBLE SUBSET is the strongest existing exactly-impartial selection rule. We run it on the same score matrices in the $(m,k)$ form. We report both its accuracy and how often it under-fills the quota. Its guarantee is the theoretical $k/(k+m)$ ratio; we know of no empirical accuracy figure on real review data.
For the language model as a standalone evaluator, the strongest reported numbers come from ReviewArena. Qwen3.6-Plus reaches a best mean absolute error of 0.700 against human scores. GLM-5.1 has the largest accept/reject gap, 1.40 points, while most models show about 0.5. Ranking all papers directly by the language model is a baseline in its own right. We expect it to lose badly at the paper level, and that loss is part of the argument.
The key ablation isolates where the calibration signal comes from. We replace the language model with a human disinterested tier: held-out human reviewer scores on a random anchor subset of $a$ papers per group, simulating senior board members with no submissions and a limited reading budget. We sweep $a$ and compare against the language model at full coverage. This tells us whether the useful ingredient is stakelessness itself, which humans on small samples may supply as well, or coverage of every paper, which only a cheap model can supply.
Why not just…?
Why not just let the language model rank every paper and drop human reviewers? A model with no submission is trivially strategyproof. But ReviewArena's reported accept/reject gap of about 0.5 points for most models, and score clumping between 4 and 7, mean paper-level rankings would be close to noise. Our design uses the model only where its errors average out. The direct-ranking baseline measures how large this difference is. If model-only ranking matched our mechanism, the human tier would be pointless, and we would report that.
Why not just z-score each group's reviewer scores and merge them into one global ranking? Standardizing removes scale shifts, but it also forces every group to have the same score distribution. That erases genuine quality differences, which is exactly the problem, and in effect reproduces proportional quotas. Letting group-level scores set quotas would reintroduce the manipulation channel, since reviewers in group A would then influence how many slots group B gets relative to A. The controlled-skew experiments show this directly: standardized merging should track the $\lambda=0$ baseline once the groups differ in quality.
Why not just have a few reviewers read papers across groups to anchor the scales? This is the most serious alternative. It uses human judgment, and a small leak of strategyproofness might be acceptable. However, each cross-reading reviewer regains some influence over their own group's slot count. How much influence depends on the number of anchors, and whether the leak is small is an empirical question. We will measure the own-paper gain available to strategic cross-readers as a function of anchor count. We will then compare that tradeoff against the stakeless human anchor ablation, which uses board members with no submissions, and against the language-model tier. If non-conflicted humans on small anchors already match the model, the recommendation becomes "staff the calibration tier with non-submitting editors." That would still be the separated-duty design, only with a human occupying the tier.
Why not just prune prolific authors further so that groups are more homogeneous? Pruning reduces component size, but it does not make groups equal in quality. Excluding more reviewers also shrinks the reviewing pool. We report group-quality variance after pruning to show whether a skew remains that is worth correcting.
Decisive Checks
The first decisive check is whether, at a realistic group size of a few hundred papers, the language model's group share $s_g$ correlates with the true group acceptance share under controlled skew after area stratification. If area-stratified correlation stays near zero, the proposal fails and we report a negative result: stakeless automated calibration does not work at current model quality.
The second check is whether the reliability-chosen $\lambda$ improves accept-set overlap over $\lambda=0$ when skew is present, without hurting it when skew is absent.
The third check is whether injected text in one group moves its quota by no more than the $\lambda k$ bound, and in practice by much less.
The fourth check covers robustness to model choice. Because ReviewArena reports that Gemma-4-26B inflates nearly all scores, we repeat the analysis with an inflating model and a discriminating one. The mechanism should survive uniform inflation, since shares are relative, but not topic-dependent inflation. Seeing which case holds tells us what property a calibration tier actually needs.
References
- On Strategyproof Conference Peer Review
- ReviewArena: A Large-Scale Cross-Conference Dataset and Benchmark for LLM Peer Review
- Impartial Peer Review
- A Group Signature Based Electronic Toll Pricing System
- PPLS: a privacy-preserving location-sharing scheme in mobile online social networks
- BEMPAS: A Decentralized Employee Performance Assessment System Based on Blockchain for Smart City Governance
- IEEE Transactions on Cybernetics publication information
- An Adaptive Responsible AI Governance Framework for Decentralized Organizations
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- ResearchTown: Simulator of Human Research Community