Peer review at a large venue runs as a chain of gates. Reviewers score a paper, an area chair (a mid-level editor who manages one topic track) writes a meta-review and ranks the track against a quota, and program chairs apply a final threshold. A paper is accepted only if it clears every gate. It is rejected if it fails any one of them. This note argues that the chain has a lopsided security property that nobody has measured. Adding tiers makes it harder to push a weak paper in, because the attacker must win at every gate. The same tiers make it easier to knock a good paper out, because the attacker needs to win at only one gate, and each new tier is one more place to do it. Torpedo reviewing, meaning a deliberate kill review against a competitor, gets cheaper as the hierarchy gets deeper.
The idea comes from a pattern in computational social choice, the formal study of voting rules. Making a chosen candidate win is often NP-hard, while making a chosen candidate lose is often solvable in polynomial time. Making someone win means beating every rival at once, which is an AND of many conditions. Making someone lose means letting any one rival overtake them, which is an OR. Editorial hierarchies build that same AND/OR structure out of their tiers rather than out of rivals. We propose to measure the resulting asymmetry on real conference data. We then test one structural fix, "two-key rejection": a rejection stands only if a second, randomly chosen path through the hierarchy agrees. This turns the attacker's OR back into an AND, and we measure what it costs in editor workload and decision quality.
Core Claim
The measure of manipulability is the smallest coalition, in number of reviewers or reviewers plus one area chair, that can flip a given paper's outcome. Let $c^-_\ell(p)$ be the smallest coalition that makes paper $p$ fail gate $\ell$, and $c^+_\ell(p)$ the smallest that makes it pass. When acceptance requires passing all $L$ gates, destructive cost is $c^-(p)=\min_\ell c^-_\ell(p)$. Constructive cost is at least $\max_\ell c^+_\ell(p)$, and usually more, because pushing a paper through one gate has to be coordinated with its standing at the others. The quantity we study is the per-paper asymmetry ratio
$$\rho(p) = \frac{c^+(p)}{c^-(p)},$$
The hypothesis has two parts. First, the ratio grows with hierarchy depth under realistic aggregation rules. Second, under two-key rejection with an unpredictable second path, destructive cost becomes roughly $c^-_{\text{path 1}}(p) + c^-_{\text{path 2}}(p)$, because the attacker cannot pick which path they will face. The ratio falls back toward 1 at a bounded cost in extra area-chair load.
We make the argument in terms of coalition size, not NP-hardness. That choice is deliberate. Control Complexity in Bucklin, Fallback, and Plurality Voting: An Experimental Approach solved NP-hard control problems on up to 32 voters in seconds with simple depth-first search, which constrains any claim that intractability protects a real review pool. The same study reports that destructive control by voter partition succeeded in nearly 100% of instances while constructive control by adding candidates often succeeded below 10% of the time. That motivates our claim: the asymmetry shows up in practice, not only in worst-case theory. We therefore compute exact minimal coalitions with integer programs, which is feasible at venue scale. Complexity results serve only as secondary characterization.
Relation to Prior Work
The theoretical pattern comes from voting-rule analyses. Complexity of Manipulating Elections with Few Candidates shows weighted constructive manipulation is NP-complete for Borda (from 3 candidates) and for Copeland and Maximin (from 4), while the destructive versions are in P. Closing the Gap of Control Complexity in Borda Elections: Solving ten open cases finds nearly all destructive control of Borda to be polynomial. Campaign Management Under Approval-Driven Voting Rules finds the same split for approval-based campaigns. Targeted Negative Campaigning: Complexity and Approximations proves destructive negative campaigning is in P across standard scoring rules. All of these study one aggregation step over competing candidates. We transfer the AND/OR argument to a composition of gates over one paper and its track, which none of them considers.
Three papers connect social choice to peer review, and each defines a gap we build on. Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments caps the probability that any reviewer is matched to any paper. On ICLR 2018 it keeps 90.8% of optimal similarity at a 0.5 cap. It explicitly does not address what happens after assignment, and it treats the venue as two flat levels. We reuse its sampler as the source of unpredictability for our second path. On Strategyproof Conference Peer Review removes self-interested downvoting by partitioning the conflict graph. It proves that stronger pairwise-consistency guarantees are incompatible with strategyproofness, and it assumes an honest aggregator. Impartial Peer Review designs a mechanism in which an applicant's own reviews cannot affect their own selection, retaining over 97% of naive selection quality. It is single-tier, and it can return fewer than $k$ winners. Both of these neutralize the incentive of reviewers who are also authors. Neither limits a colluder with no submission in the pool, and neither limits a compromised area chair.
The human-behavior evidence in A Dataset on Malicious Paper Bidding in Peer Review shows that low-effort bidding manipulation reached a 100% success rate at landing on a target paper, and that low-rank anomaly detection did no better than chance. It gives us a realistic model of how attackers reach a paper. Its setting was 31 participants in a mock single-tier conference.
The organizational literature supplies motivation rather than method. Dilemmas and Governance Strategies in the Implementation of the Modern Apprenticeship System in Chinese Vocational Colleges: A Perspective from Principal-Agent Theory argues that each handover layer opens an information gap an intermediate agent can exploit. That is why our threat model lets the area chair be compromised.
Research Program
We build a simulator of a three-tier venue in which each tier's rule is a plug-in. At the reviewer tier, the paper-level rules are the mean, the median, and a veto rule in which any strong-reject score forces a low meta-score. At the area-chair tier, each track ranks its papers against a quota. At the program-chair tier, a global threshold is applied to calibrated scores. Depth is a controlled variable: we test flat (reviewers straight to a global top-$k$), two-tier, and three-tier, and a four-tier variant with senior area chairs.
For every paper and every configuration, an integer program computes $c^-(p)$ and $c^+(p)$. The coalition is allowed to include assigned reviewers and, in a second threat model, the paper's area chair. Scores can move anywhere within the rubric. Coalitions may be constrained to reviewers who can realistically reach the paper under a given assignment rule. We use the ICLR 2017 co-authorship components as a proxy for who is likely to collude.
On the theory side, we will prove the min/max composition bounds for threshold-and-quota hierarchies. We will also characterize when a single-tier rule is itself asymmetric: quota ranking inherits the OR structure, because any rival in the track can displace the paper. Finally, we will prove the additive destructive-cost bound for two-key rejection when the second path is drawn by the randomized assignment sampler with cap $Q$. The attacker's chance of controlling both paths is at most the product of the per-path caps, and the proof follows that product structure.
Two-key rejection works as follows. Any paper that would be rejected, but whose reviewer-tier scores sit above a trigger band, receives one additional reviewer and a second area chair. Both are sampled with probability caps. The paper is rejected only if both paths reject.
The band is the main design knob, and it matters most. It determines extra workload, which we report as additional reviews per submission. It also determines how much honest noise is corrected versus introduced.
Baselines
The naive reference point is the status quo: deterministic similarity-maximizing assignment, mean score aggregation, and a three-tier quota hierarchy. We report its distribution of $\rho(p)$ and the fraction of accepted papers that a coalition of one or two reviewers could flip to rejection. Any fix must lower that fraction to count as progress.
The strongest existing competitor for this question is randomized assignment with a 0.5 cap from Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments, combined with median aggregation. The first component is the best published defense against attackers reaching a paper. The second is the obvious robust-statistics defense against a single outlier. Its reported number is 90.8% of optimal sum-similarity on ICLR 2018 at the 0.5 cap. No reported result was found on Lacuna for coalition-size or torpedo-success metrics under this baseline, so we compute those ourselves. The comparison is fair because both our method and this baseline use the same similarity matrix, the same sampler and cap, and the same honest-score model. Both also receive the same total reviewer budget: this baseline gets the extra reviews that two-key rejection spends, added as a fourth reviewer on every borderline paper. Two-key rejection beats it if, at equal review budget, it raises the median destructive coalition size on accepted papers, or lowers the one-reviewer-flip fraction, without increasing constructive success or honest ranking error.
For partitioned venues we also run On Strategyproof Conference Peer Review on the ICLR 2017 conflict graph. That paper reports that removing 100 high-degree authors shrinks the largest paper component from 133 to 8. The baseline tests whether removing the incentive to torpedo already removes the practical attack surface.
Two ablations isolate what matters. The first keeps two-key rejection but assigns the second path deterministically. If most of the gain disappears, the fix works through unpredictability, not redundancy. The second spends the same extra reviews on randomly chosen papers instead of would-be rejections. If that matches our method, the AND structure is not the active ingredient.
Why not just…?
Why not just use median or trimmed-mean scores? This is the most serious threat. With three reviewers the median ignores one torpedo, which may capture most real attacks. It does not touch the area-chair tier, where one compromised meta-reviewer is a single-person OR. It also fails once the coalition holds a majority of a small panel. We run median as part of the strongest baseline and report $\rho$ separately for the reviewer-only and area-chair-inclusive threat models. If median alone closes the gap under the reviewer-only model, we will say so and narrow the claim to the area-chair tier.
Why not just randomize assignments? Randomization bounds who reaches a paper, not what a reached reviewer does. The 100% bidding-success result in A Dataset on Malicious Paper Bidding in Peer Review shows that reaching a target is cheap without caps. With caps, any reviewer who is legitimately assigned still holds the full destructive lever. The deterministic-second-path ablation measures how much randomization adds on top of redundancy.
Why not just partition the conflict graph so no one has a reason to torpedo? Partitioning removes the author-reviewer incentive only. It leaves colluders acting for a friend's competing paper, and it assumes the aggregator is honest. Its own impossibility result also forfeits pairwise consistency across groups. The Divide-and-Rank baseline, run under our collusion model, shows how much attack surface remains.
Why not just flatten the hierarchy? If depth causes the asymmetry, removing tiers should reduce it, and we test that directly as the flat configuration. Flattening, however, returns thousands of papers to one decision-maker, so we report its honest ranking error alongside its $\rho$.
Proposed Experiments
Honest scores are simulated from latent paper quality plus reviewer noise. The noise is calibrated to the score dispersion in the ICLR portion of A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications. Assignments come from the ICLR 2018 matching data of 2,435 reviewers and 911 papers. Coalition structure comes from the ICLR 2017 conflict graph. For a journal-style multi-round hierarchy, where rounds act as tiers, we replicate the analysis on the round structure of FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes.
The first decisive check is whether median destructive coalition size on accepted papers falls, and $\rho$ rises, as depth goes from flat to four tiers under the status quo. If $\rho$ does not grow with depth, the central claim is false.
The second check is whether two-key rejection at equal review budget beats the randomized-plus-median baseline on one-reviewer and area-chair flip rates. Honest ranking error, measured as Kendall correlation with latent quality and top-$k$ precision, must stay within one standard error of that baseline. This check produces the trigger-band curve of workload against protection.
The third check is whether the random-reallocation ablation underperforms our method. If it does not, the conjunctive structure is not doing the work, and we would report redundant review as the practical recommendation instead.
As a practical-tractability check in the spirit of the voting experiments, we also report solver time for the attacker's integer program. Our argument should rest on coalition size, not on attackers being unable to compute an attack.
References
- Control Complexity in Bucklin, Fallback, and Plurality Voting: An Experimental Approach
- Complexity of Manipulating Elections with Few Candidates
- Closing the Gap of Control Complexity in Borda Elections: Solving ten open cases
- Campaign Management Under Approval-Driven Voting Rules
- Targeted Negative Campaigning: Complexity and Approximations
- Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments
- On Strategyproof Conference Peer Review
- Impartial Peer Review
- A Dataset on Malicious Paper Bidding in Peer Review
- Dilemmas and Governance Strategies in the Implementation of the Modern Apprenticeship System in Chinese Vocational Colleges: A Perspective from Principal-Agent Theory
- A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes