Conferences routinely correct for reviewers who are systematically harsh or lenient before making accept/reject decisions. On open platforms where scores and decisions are published, this correction leaks information. An outsider who sees that a paper with a low average score was accepted can work backward to guess which reviewer the chairs discounted, and so who reviewed the paper. The only published remedy has the chairs deliberately make some wrong decisions on borderline papers so that the guess becomes unreliable. We propose a different place to put the privacy: in the shape of the editorial hierarchy itself. That means how reviewers are grouped under Area Chairs, which tier performs the correction, and what each tier releases. Choosing these well should make many different reviewer assignments equally consistent with what the public sees, without making any decision worse on purpose.
The second idea comes from the literature on privacy as an adoption barrier. Contact-tracing studies show that when people fear exposure, they opt out, and the system then starves for data. Peer review has the same loop in a quieter form. Reviewers who believe they can be identified do not refuse to review. They soften their scores toward the middle, and that destroys exactly the signal that correction relies on. We therefore evaluate editorial designs on decision accuracy after reviewers react to their own identification risk. This is a comparison no existing work makes. It could reverse the apparent ranking of designs.
Core Claim
Calibration with Privacy in Peer Review defines two quantities. Conference error $E_C$ is the probability of accepting the lower-quality paper. Adversary error $E_A$ is the probability that a maximum-a-posteriori attacker fails to recover reviewer assignments. That paper proves a trade-off frontier between them, and on that frontier privacy is bought with randomized, suboptimal decisions. Its analysis uses a two-reviewer, two-paper setting and leaves open whether multi-tier delegation could provide privacy structurally.
Our claim is that, at realistic conference scale, the editorial structure is itself a lever on $E_A$. The relevant choices are the overlap pattern of reviewer–paper assignments inside an Area Chair's block, the block size, and whether correction is applied per paper or through block-level acceptance quotas. With a well-chosen structure, a design can reach the $E_A$ of the randomized mechanism at strictly lower $E_C$.
We further claim the comparison changes once reviewers respond to risk. We model reviewer $i$'s reported score as
$$s_{ij} = \lambda(\pi_i)\, q_{ij} + (1-\lambda(\pi_i))\, \bar{s} + b_i + \epsilon_{ij},$$
where $q_{ij}$ is the reviewer's honest assessment of paper $j$, $\bar{s}$ is the scale midpoint, $b_i$ is the reviewer's bias, $\epsilon_{ij}$ is noise, $\pi_i$ is the reviewer's perceived identification probability (one minus $E_A$ under the deployed design), and $\lambda \in [0,1]$ is a decreasing "candor" function. The structure of this equation is the peer-review analogue of the adoption loop in Privacy, adoption, and truthful reporting. That paper models people as weighing private benefit against privacy cost, which reduces both participation and representativeness. Here, perceived risk reduces informativeness instead of participation.
Why This Intersection, and What Is Already Done
The adoption-barrier literature motivates the problem. Digital contact tracing: technologies, shortcomings, and the path forward shows that utility collapses non-linearly when trust fails. Beyond legislation and technological design and the cross-country acceptability survey show that perceived trust in the governing institution, more than technical guarantees, drives participation. Together they warn that an editorial privacy mechanism matters only through how reviewers perceive it. For that reason we measure behavior under perceived risk, not just formal $E_A$.
The editorial-structure literature supplies the design space. Journal front matter such as IEEE Transactions on Cybernetics publication information documents the standard Editor-in-Chief → Associate Editor → reviewer tiering. Paper Copilot documents how this tiering has scaled, from 490 submissions and 31 Area Chairs at ICLR 2017 to 11,672 submissions and 823 Area Chairs at ICLR 2025. It also reports falling "decision entropy," meaning Area Chairs increasingly threshold on mean scores rather than exercise judgment. This trend matters to us because uncalibrated thresholding is the de facto current baseline.
The closest existing work is Calibration with Privacy in Peer Review. We differ from it in three ways. We treat the hierarchy and assignment graph as design variables rather than fixed inputs. We work at the scale of thousands of papers rather than in the stylized two-by-two case. We make reviewer behavior endogenous to privacy.
On the Privacy-Utility Tradeoff in Peer-Review Data Analysis addresses a neighboring problem: releasing reviewer statistics after the fact using Laplace noise projected onto constraints from public score multisets. We reuse two things from it, its Beta-distributed reviewer simulation and its insight that public score multisets act as a structural scaffold. We apply that insight to the decision mechanism rather than to data release. Private Hierarchical Governance for Encrypted Messaging is a conceptual precedent: it shows that tiered moderation can be arranged so that a higher tier acts without seeing lower-tier content. It addresses encryption, not inference from released outputs, so it informs our framing but not our method.
Research Program
The first component is a scalable attacker. We extend the maximum-a-posteriori adversary of Ding et al. from the two-by-two case to full conference graphs. Given published raw scores, decisions, and Area Chair groupings, the attacker searches for the reviewer assignment and bias vector that best explain the observed decisions under a known calibration rule. We implement it as an integer program, with a Gibbs-sampling fallback for large blocks. We report $E_A$ both as exact-assignment failure and as per-review identification probability, because individual reviewers care about the second.
The second component is the design space of editorial structures. One design variable is block size $k$, the number of reviewers pooled under one Area Chair or Senior Area Chair. A second is the assignment overlap pattern within a block. This ranges from disjoint reviewer sets to designed overlaps, in which several reviewers each cover several papers so that many assignments produce the same calibrated outcomes. A third is the locus of calibration: per-paper adjustment versus block-level quotas, where the Area Chair receives calibrated rankings but only decisions are released. A fourth is which tier releases what.
For each design we trace the $(E_C, E_A)$ curve. We also search for structures by optimizing the assignment graph for ambiguity, subject to reviewer-expertise constraints, using a matching objective with an added term that penalizes uniquely identifying overlaps.
The third component is the endogenous-candor model. We close the loop by iterating: deploy a design, compute each reviewer's $\pi_i$, update $\lambda(\pi_i)$, regenerate scores, and recompute $E_C$ until a fixed point is reached. We require one empirical input, the shape of $\lambda$. To estimate it, we will run a vignette study with reviewers recruited from ML venues. Participants score short, fixed paper summaries under randomized stated identification risks of 0%, 10%, 30%, and 60%. We measure how much their scores shrink toward the midpoint. Paper Copilot's survey of over 1,800 authors, roughly 60% of whom were willing to share anonymized review records, shows that such populations engage with privacy questions. It is not evidence about candor itself. If the vignette study finds no shrinkage, the endogenous component drops out and the paper reduces to the structural claim, which still stands on its own.
Baselines
The naive reference point is uncalibrated mean-score thresholding within each Area Chair block. This is the regime Paper Copilot documents conferences drifting toward. Because it applies no correction, there is nothing to invert. Its $E_A$ is therefore maximal with respect to the calibration channel, and its $E_C$ reflects the full cost of reviewer bias. Every proposed design must beat its $E_C$ while keeping $E_A$ within a stated tolerance of it.
The second reference point is full calibration with published raw scores and decisions. It achieves the lowest $E_C$ when reviewers are candid, and it measures how much the attacker can extract when nothing is hidden.
The strongest competitor is the noise-robust randomized calibration of Calibration with Privacy in Peer Review, because it is the only published mechanism that explicitly targets the same trade-off. That paper reports from simulations with 100 reviewers and 100 papers that exogenous calibration gives a "multi-fold reduction in error compared to no calibration, even under strict privacy constraints." It reports no specific numerical $E_C$ or $E_A$ values; there is no reported result on Lacuna for those figures. We will therefore reimplement it and run it on identical simulated conferences.
The simulated conferences will use the same reviewer bias distributions and Beta-distributed quality model as On the Privacy-Utility Tradeoff in Peer-Review Data Analysis. They will use the same exogenous reputation signal and an assignment-solver budget matched across designs. We will sweep its randomization parameter to trace its full frontier rather than tune a single point. Beating it means one of two things: a lower $E_C$ at every $E_A$ level on the matched frontier, or, under the endogenous-candor model, a higher fixed-point decision accuracy.
Two ablations isolate the key ingredients. The first keeps the block hierarchy and the release policy but uses a standard expertise-only assignment, with no ambiguity term. This shows whether gains come from graph design or merely from pooling. The second runs every design with $\lambda \equiv 1$ (fully candid reviewers). This shows how much of any ranking reversal is due to endogenous behavior.
Why not just…?
Why not just publish decisions only and keep scores private? At closed venues this is already the norm. On the Privacy-Utility Tradeoff in Peer-Review Data Analysis documents the resulting data drought for metascience, so it is a real policy cost. It also does not fully close the channel. Decisions combined with author-shared scores, which are exactly what Paper Copilot crowdsources for closed venues, still let an attacker partially invert calibration. We include "decisions plus a random fraction of leaked scores" as a release condition. This lets us measure how much protection closure actually buys.
Why not just stop calibrating, as conferences seem to be doing? This is a serious threat. If reviewer bias at the thousands-of-papers scale averages out within blocks, uncalibrated thresholding could sit near the frontier at no cost. Ding et al.'s qualitative claim of multi-fold error reduction from calibration argues against this, but it comes from a 100-paper simulation. Our naive baseline tests it directly at realistic scale. If the $E_C$ gap closes, we will report that calibration is not worth its privacy cost, and that would itself be a useful finding for program chairs.
Why not just use Ding et al.'s randomized calibration at scale? Its cost, deliberately wrong decisions on borderline papers, is borne by authors and is hard to defend institutionally. That cost grows with the borderline mass, which Paper Copilot's falling decision entropy suggests is large. Whether the structural designs beat it is the central experiment, not an assumption. If they do not, the endogenous analysis still asks which of the two mechanisms reviewers trust more, measured via the vignette $\lambda$.
Why not just add differential-privacy noise to the published scores? The projection method of On the Privacy-Utility Tradeoff in Peer-Review Data Analysis shows that noisy releases can stay useful. However, noisy published scores alongside exact decisions leave the calibration inference intact, because the adversary can attack the decisions. We include score-level Laplace noise plus projection as an additional comparison and expect it to leave $E_A$ largely unchanged.
Proposed Experiments
All experiments run on simulated conferences scaled to Paper Copilot's figures: about 11,700 papers, about 820 Area Chairs, and reviewer loads drawn from the ICLR 2024–2025 score-multiset distributions that dataset provides. We also run 1,000-paper and 100-paper scales; the 100-paper scale lets us check our reimplementation against Ding et al.'s qualitative result. For each design we report the $(E_C, E_A)$ frontier, per-review identification probability, the fixed-point $E_C$ under the estimated candor function, and attacker runtime.
The decisive check is the comparison against the randomized-calibration baseline. If some structural design reaches the same $E_A$ at lower $E_C$ at full scale, and the result survives the expertise-only ablation, the structural claim holds.
A second decisive check concerns endogenous behavior. If, under the measured $\lambda$, full transparent calibration falls below uncalibrated thresholding in fixed-point accuracy, privacy is demonstrated to be a precondition for calibration's value rather than a cost against it. This would carry the adoption-barrier argument from contact tracing into editorial design.
A robustness check varies the attacker's knowledge: known versus estimated bias distribution, and known versus unknown block membership. This guards against designs that are secure only against a weak attacker.
References
- Calibration with Privacy in Peer Review
- Privacy, adoption, and truthful reporting: A simple theory of contact tracing applications
- Digital contact tracing: technologies, shortcomings, and the path forward
- Beyond legislation and technological design: The importance and implications of institutional trust for privacy issues of digital contact tracing
- Acceptability of App-Based Contact Tracing for COVID-19: Cross-Country Survey Study
- IEEE Transactions on Cybernetics publication information
- Paper Copilot: Tracking the Evolution of Peer Review in AI Conferences
- On the Privacy-Utility Tradeoff in Peer-Review Data Analysis
- Private Hierarchical Governance for Encrypted Messaging