When Reviews Gate Access: Collusion, Sabotage and the Scale of Peer Review in the AI Submission Surge

Yuchen Li, Xintong Li, Ken-Tye Yong, Nguyen H. TranarXiv 2026 · arXiv

TL;DR

As AI-driven submission surges force major conferences to adopt "gated" peer review—where reviewing is mandatory and eligibility is earned through past success—reviewers have effectively become competitors. This paper uses agent-based modeling to show that while large-scale conferences are theoretically safer from collusion, current bidding and eligibility rules create a "scale paradox" that empowers collusion rings and allows incumbents to sabotage newcomers.

What Problem Does This Paper Address?

The "AI submission surge" has pushed venues like NeurIPS and ICLR to their breaking points, with tens of thousands of papers submitted per cycle. To manage this, organizers have introduced new rules: submission caps, mandatory reciprocal reviewing, and "qualified" reviewer pools (only those with previously accepted papers can review).

The authors argue these rules fundamentally change the incentives of peer review. When your chance to submit next year depends on your paper being accepted this year, and you are reviewing the very people you are competing with for a limited number of "acceptance slots," the system invites two types of deviance:

  1. Sabotage: Lowering a competitor’s score to increase your own relative standing or gatekeep them out of the reviewer pool.
  2. Collusion: Forming "rings" to artificially boost the scores of allies.

The paper seeks to understand how these behaviors interact, how they scale with conference size, and how they impact the fairness and quality of the accepted paper list.

Methods

The researchers employed a multi-pronged approach:

  • Numerical Agent-Based Modeling (ABM): They built a simulated conference environment calibrated to public statistics from NeurIPS and ICLR. They ran 6,660 simulated conferences with up to 10,000 "numerical agents" programmed to optimize their acceptance rates while accounting for the risk of being caught by audits.
  • Large Language Model (LLM) Agents: To test behavioral nuances that simple math might miss, they ran 90 conferences using agents driven by DeepSeek V4.1 and Qwen 3.7. These agents were given the rules and payoffs and asked to make decisions in natural language.
  • Mathematical Proofs: They derived the "Exposure Law" to define the probability of a paper encountering a colluder under different assignment regimes.

Datasets

The paper does not use a traditional training dataset. Instead, it uses:

  • Calibration Data: Publicly available metadata and consistency experiment results from NeurIPS 2021 and ICLR to ensure the simulation's "honest" baseline matches real-world behavior (e.g., a 27% acceptance rate).
  • Simulated Benchmarks: A newly introduced testbed for "multi-cycle" conference dynamics, tracking how alliances and eligibility evolve over 20 rounds (simulating 2-3 years of real-world cycles).

Key Insights and Ideas

  • The Sabotage-Collusion Feedback Loop: Sabotage and collusion aren't just additive; they amplify each other. Sabotage pushes a high-quality paper below the acceptance threshold, freeing up a "seat." Collusion then ensures that seat is filled by an ally’s paper.
  • The Scale Paradox: Under perfectly random reviewer assignment, growing the conference size ($A$) protects the system because the chance of meeting an ally ($E = Rm/A$) drops. However, reviewer bidding (where reviewers request specific papers) creates a "floor." In a pool of 10,000, a 20-member ring wins only 0.24% of its papers via random assignment but a staggering 71% when 30% of their bids are granted.
  • The Incumbency Loop: Because eligibility to review is often tied to having accepted papers, sabotage becomes a weapon for incumbents. By sabotaging newcomers, incumbents keep them out of the reviewer pool, ensuring that only "established" players (who may be part of existing rings) hold the power to review.
  • LLM "Tribalism": The LLM agents exhibited "in-group bias" that went beyond rational payoff optimization. Ring members sabotaged outsiders significantly more than was strictly necessary to help their allies, suggesting that social structures in peer review might breed hostility toward "the out-group."

Key Results

  • Signal vs. List: Collusion primarily changes who is accepted (fairness), whereas sabotage erodes the signal (the correlation between scores and actual paper quality).
  • The Power of Caps: Limiting the number of "allies" an agent can have to "social brain" layers (like Dunbar’s number of ~15 or ~50) keeps collusion nearly harmless. Without these caps, agents eventually ally with half their research area, causing the "accepted value" (total quality) of the conference to fall by 35%.
  • Bidding Risks: The authors provide a design rule for conference chairs: to keep a 20-member ring's success rate below 10%, the "grant rate" for bids must stay below 0.55%.

Limitations and Future Work

  • Audit Assumptions: The model assumes a 1% audit rate. In reality, detecting subtle sabotage (giving a 5 instead of a 7) is much harder than detecting blatant collusion.
  • Static Quality: The model assumes paper quality is independent of the review system. In reality, if the system is seen as "rigged," high-quality authors might stop submitting altogether.
  • Future Directions: The authors suggest future work should focus on "matching quality"—finding the sweet spot where bidding helps find expert reviewers without opening the door to collusion rings. They also aim to calibrate scores against real-world data from platforms like OpenReview.

Summary updated

Abstract

AI tools have helped push submissions to the largest machine-learning venues to tens of thousands per cycle. Venues have answered with rules that tie access to review: per-author caps, mandatory reciprocal reviewing, reviewer eligibility earned through accepted papers, and assignment based on reviewer bids. Under these rules a reviewer is also a competitor of the authors. We study two deviations that the rules make attractive. Sabotage lowers a competitor's score. Collusion raises an ally's score. We build an agent-based conference calibrated to public statistics of NeurIPS and ICLR and run 6,220 simulated conferences. Four results stand out. First, the two deviations do different harm and amplify each other. Collusion changes who is accepted, sabotage erodes what scores say about quality, and rings that may also sabotage gain more than rings that only collude. Second, under random assignment the chance that collusion changes a decision is set by one exposure index, $E = Rm/A$, the expected number of allies among a paper's reviewers. Growth therefore protects random assignment, but bidding removes the protection. In a pool of 3,000 reviewers a 20-member ring turns 0.8% of its papers into acceptances under random assignment and 71% when 30% of allies' bids are granted. Third, when agents choose their allies every cycle, caps at the inner layers of the social brain keep collusion nearly harmless. Without a cap agents ally with half of their area and accepted value falls by 35%. Fourth, reviewer eligibility tied to acceptances turns sabotage into a weapon of incumbents, and a longer eligibility window widens the gap between incumbents and newcomers only when sabotage is possible.
BibTeX
@misc{li2026when,
  author = {Yuchen Li and Xintong Li and Ken-Tye Yong and Nguyen H. Tran},
  title = {{When Reviews Gate Access: Collusion, Sabotage and the Scale of Peer Review in the AI Submission Surge}},
  year = {2026},
  eprint = {2611.19054},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19054},
}
Figure