Stake-Free Bridges: Calibrated, Strategyproof Peer Review Through the Editorial Tier

Conference peer review has two known failure modes that current fixes cannot address together. Reviewers who are also submitting authors can lower their scores for competing papers to help their own. Honest reviewers also use very different scales: one reviewer's 6 is another's 4. The best defense against the first problem divides authors and reviewers into groups so that no one ever reviews a paper that competes with their own. The best defense against the second compares reviewers through shared papers, which requires the opposite: a review graph dense enough that every pair of reviewers can be linked through papers they have both scored. Put simply, strategyproofness wants the graph cut apart and calibration wants it knitted together.

We propose to use the editorial board as the stitching. Area chairs, senior area chairs, and other evaluators who have no submission in the ranking they touch can review a small, carefully placed set of papers that cross the partition. Their scores are used only to align scoring scales between groups and to decide how many acceptance slots each group receives. They never set the ranking of papers inside a group. Because these evaluators have no competing paper, what they report cannot help their own position. Because they bridge the groups, the global ranking regains comparability across groups. The concrete questions are how few such bridge reviews are needed, where they should go, and how much precision they recover. We also ask how the design degrades when the bridge evaluators have their own biases, such as favoring their sub-area.

Core Claim

A partitioned, strategyproof review process plus a small budget of stake-free "bridge" reviews from the editorial tier can recover most of the ranking precision of full-graph calibration run on honest reviewers. It keeps the zero-manipulation-gain guarantee for submitting reviewers. The claim is new in two respects. It treats the editorial hierarchy as a structural ingredient of the mechanism rather than as an administrative layer. It also turns a known contradiction between two results into a design problem with a measurable budget. The two results are the partition requirement of On Strategyproof Conference Peer Review and the double-connectivity requirement of Least Square Calibration for Peer Reviews.

Where the Prior Work Stands

Divide-and-Rank (On Strategyproof Conference Peer Review) splits the author–reviewer conflict graph so that group A reviews group B and vice versa. It merges the groups with a sorting step that satisfies a weak unanimity property, and it proves that a stronger pairwise unanimity is incompatible with strategyproofness. It motivates this proposal: its own analysis leaves cross-group comparability open, and its use of ICLR-17 shows that real conflict graphs can be broken apart by removing about 7% of prolific authors.

Least Square Calibration (Least Square Calibration for Peer Reviews) jointly estimates paper quality and each reviewer's scoring function. It proves that exact recovery in the linear case needs a doubly connected reviewer–paper graph, meaning reviewers must be linked through at least two shared papers. It supplies our calibration method and also states the constraint our bridges must satisfy. It assumes honest reviewers, which we do not.

Impartial selection (Impartial Peer Review) takes a different route to strategyproofness. It picks winners randomly from a "credible" set, which guarantees that a reviewer's ratings never affect their own selection. The price is an approximation factor of $k/(k+m)$, where $k$ is the number of papers accepted and $m$ is the number of reviews each reviewer writes, and the mechanism may accept fewer than $k$ papers. We treat it as a competing strategyproof baseline. Like Divide-and-Rank, it is single-tier.

Randomized assignment (Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments) caps the probability that any reviewer lands on any paper. It supplies the tool we use to keep bridge evaluators from being steered onto particular papers. It does not address strategic scoring, which is the gap our partition covers.

The detection literature constrains the design. On the Detection of Reviewer-Author Collusion Rings From Paper Bidding and A Dataset on Malicious Paper Bidding in Peer Review both show that finding colluders after the fact is unreliable, which argues for guarantees built into the structure.

On the editorial side, the file's governance sources describe the tiered structure. The IEEE publication-information pages such as IEEE Transactions on Cybernetics publication information cover editors-in-chief, associate editors, and reviewers. Principal–agent analyses such as Dilemmas and Governance Strategies in the Implementation of the Modern Apprenticeship System in Chinese Vocational Colleges argue that intermediate layers carry their own incentives. None of these sources gives a mechanism. They justify treating area chairs as agents with private preferences, not as neutral pipes.

Research Program

The mechanism has three layers.

Partitioning and within-group ranking. Submitting reviewers are partitioned as in Divide-and-Rank, and only reviewers from the other group rank the papers in each group. As a result, a submitting reviewer's report can affect neither the within-group order of their own paper nor anything else about it.

Bridge reviews. A stake-free tier writes a budget of $B$ bridge reviews on papers from both groups. This tier consists of area chairs and reviewers with no submission in either group being merged.

Merging. A calibration model is fit on the full graph that the bridges connect. We use linear LSC first, with monotone variants as a follow-up. From this fit the mechanism computes only two things: a common score scale, and a slot split $(k_A, k_B)$ with $k_A + k_B = k$. The groups' internal orders are then interleaved according to the calibrated scores.

Strategyproofness for submitting reviewers holds by construction. Their reviews only touch the other group's internal order, which, as in Divide-and-Rank, cannot move their own paper. They also feed the scale-alignment step, which we therefore anchor using bridge scores alone. The first theoretical deliverable is to prove this carefully. That includes the subtle point that submitting reviewers' scores must not enter the estimate of the slot split.

The second theoretical deliverable is a bound on the bridge budget. In the noiseless linear model, each reviewer's score is $y_{rp} = a_r x_p + b_r$, where $a_r$ and $b_r$ are that reviewer's scale and offset. Restoring double connectivity between two components requires a bridge structure that ties reviewers in both components to at least two shared papers. That suggests a constant number of bridge reviewers per component pair when noise is absent. Under noise, the error in estimating the relative offset between groups should shrink roughly as $\sigma/\sqrt{B}$, because it is an average of $B$ independent noisy comparisons. We will test whether the ranking error follows this scaling.

The third deliverable is a bridge placement rule. Bridges go preferentially to papers near each group's likely acceptance boundary, because errors there change decisions.

The fourth deliverable handles strategic bridge evaluators. An area chair who favors a sub-area can tilt the slot split. We bound this influence in two ways. We cap each bridge evaluator's assignment probabilities using the randomized assignment linear program. We estimate the split with a median over bridge evaluators rather than a mean. The goal is a bound on how far any single evaluator can move $k_A$.

Baselines

The naive reference point is plain score averaging over all reviews, which Impartial Peer Review calls VANILLA. In the noisy synthetic ICLR 2019 setting at noise level $\sigma = 0.5$, Least Square Calibration for Peer Reviews reports about 39% top-paper precision for averaging and about 71% for a Bayesian baseline. Averaging also gives manipulators full leverage, so we report both its precision and its manipulation gain.

The strongest calibration competitor is LSC run on the full connected graph with honest reviewers. It reaches 80.1% precision at $\sigma = 0.5$ on synthetic ICLR 2019 and 82.2% on the Hamburg peer-grading data (Least Square Calibration for Peer Reviews). We treat it as a ceiling on precision rather than as something to beat. It is not strategyproof. We will also run it with a fraction of strategic reviewers, where its precision should fall. Beating LSC in that strategic setting, and coming within a few points of its honest-reviewer precision, would count as success.

The strongest strategyproof competitors are two.

The first is Divide-and-Rank with slots split in proportion to group size. No precision number is reported on Lacuna for Divide-and-Rank. The file only reports graph-partitionability statistics on ICLR-17, so we will reimplement it.

The second is CREDIBLE SUBSET, reported to capture "over 97%" of the quality of standard selection at IJCAI 2013 scale (Impartial Peer Review). We will measure it under our miscalibrated-reviewer simulator and report how often it accepts fewer than $k$ papers. Its reported figure assumes calibrated scores.

For fairness, all methods receive the same simulated reviewer population, the same review load per paper, and the same total number of reviews. The bridge reviews come out of our budget rather than being added on top. Every method is tuned on held-out random seeds with an equal number of tuning configurations.

We run two ablations to isolate the key ingredient. The first draws bridges from submitting reviewers instead of stake-free ones, which should restore precision but reintroduce measurable manipulation gain. The second places bridges at random instead of at acceptance boundaries.

Why not just…?

Why not just use proportional slot quotas in Divide-and-Rank? This is the most serious threat. If groups were large random splits of the paper pool, their quality distributions would match, and a proportional split would already be nearly optimal. Partitions built from conflict graphs are not random, though. They follow co-authorship components, which cluster by topic and lab, and the ICLR-17 analysis in On Strategyproof Conference Peer Review shows components of very uneven size. We will measure directly how much precision a proportional split loses against a split computed from true quality, on partitions derived from ICLR-17. If that gap is negligible, the proposal's value narrows to scale calibration, and we will report that plainly.

Why not just run full-graph calibration with randomized assignment caps? Randomized assignment (Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments) limits who can reach which paper. It does nothing to stop an assigned reviewer from lowering a competitor's score, and LSC explicitly assumes honesty. The manipulation-gain experiment below measures what this combination leaves exposed.

Why not just detect manipulators and discard their reviews? The best dense-subgraph detector reaches only 31% Jaccard similarity to true colluders, and colluders get assigned undetected in a "safe zone" where 54% obtain a target paper (On the Detection of Reviewer-Author Collusion Rings From Paper Bidding). Score-level downgrading of competitors is plausibly even harder to separate from honest disagreement.

Why not just let non-submitting reviewers do everything? Where they exist in sufficient number, this works, and our mechanism reduces to it. The practical question is supply: at ML conferences most reviewers submit papers. We will measure the stake-free fraction on data where reviewer and author identities can be linked. We will then report the smallest stake-free fraction at which our method matches the full-honesty LSC ceiling.

Proposed Experiments

We use the ICLR-17 conflict graph (489 submissions, 1,417 authors) to generate realistic partitions and component structure. The ICLR 2018 assignment data (2,435 reviewers, 911 papers) provides similarity-based review graphs. On top of these we simulate paper qualities and reviewer scoring functions following the synthetic ICLR 2019 protocol of Least Square Calibration for Peer Reviews, with linear, monotone, and concave reviewer types and noise $\sigma \in \{0.25, 0.5, 1.0\}$. The Hamburg peer-grading data serves as a real-score check with no partition structure. There we impose artificial partitions and measure how much bridging recovers relative to the unpartitioned LSC figure of 82.2%.

The metrics are three:

  • top-$k$ precision against true quality;
  • manipulation gain, defined as the largest improvement in own-paper rank that a submitting reviewer can obtain by best-responding, and estimated by search over score vectors;
  • the bridge budget $B$, expressed as the percentage of total reviews.

Strategic behavior follows the rival-tanking model of Impartial Peer Review, with the strategic fraction set to 0, 10, and 30% of submitting reviewers. Separately, we inject biased bridge evaluators with a fixed sub-area preference to test the median-plus-caps defense.

The results can be read as follows:

  • Success. On ICLR-17-derived partitions under noise $\sigma = 0.5$, a bridge budget near 5% of reviews recovers most of the gap between proportional-split Divide-and-Rank and the honest LSC ceiling, while manipulation gain for submitting reviewers stays exactly zero.
  • Theory check. Slot-split error should fall roughly as $1/\sqrt{B}$, with boundary-targeted bridges beating random ones at equal budget.
  • Refutation. Either the proportional-quota gap turns out negligible on real partitions, or a single biased area chair can shift the slot split enough to change more acceptances than the bridges correct. In either case the idea fails, and we would report which one happened.