TL;DR
As AI-driven submission surges force major conferences to adopt "gated" peer review—where reviewing is mandatory and eligibility is earned through past success—reviewers have effectively become competitors. This paper uses agent-based modeling to show that while large-scale conferences are theoretically safer from collusion, current bidding and eligibility rules create a "scale paradox" that empowers collusion rings and allows incumbents to sabotage newcomers.
What Problem Does This Paper Address?
The "AI submission surge" has pushed venues like NeurIPS and ICLR to their breaking points, with tens of thousands of papers submitted per cycle. To manage this, organizers have introduced new rules: submission caps, mandatory reciprocal reviewing, and "qualified" reviewer pools (only those with previously accepted papers can review).
The authors argue these rules fundamentally change the incentives of peer review. When your chance to submit next year depends on your paper being accepted this year, and you are reviewing the very people you are competing with for a limited number of "acceptance slots," the system invites two types of deviance:
- Sabotage: Lowering a competitor’s score to increase your own relative standing or gatekeep them out of the reviewer pool.
- Collusion: Forming "rings" to artificially boost the scores of allies.
The paper seeks to understand how these behaviors interact, how they scale with conference size, and how they impact the fairness and quality of the accepted paper list.
Methods
The researchers employed a multi-pronged approach:
- Numerical Agent-Based Modeling (ABM): They built a simulated conference environment calibrated to public statistics from NeurIPS and ICLR. They ran 6,660 simulated conferences with up to 10,000 "numerical agents" programmed to optimize their acceptance rates while accounting for the risk of being caught by audits.
- Large Language Model (LLM) Agents: To test behavioral nuances that simple math might miss, they ran 90 conferences using agents driven by DeepSeek V4.1 and Qwen 3.7. These agents were given the rules and payoffs and asked to make decisions in natural language.
- Mathematical Proofs: They derived the "Exposure Law" to define the probability of a paper encountering a colluder under different assignment regimes.
Datasets
The paper does not use a traditional training dataset. Instead, it uses:
- Calibration Data: Publicly available metadata and consistency experiment results from NeurIPS 2021 and ICLR to ensure the simulation's "honest" baseline matches real-world behavior (e.g., a 27% acceptance rate).
- Simulated Benchmarks: A newly introduced testbed for "multi-cycle" conference dynamics, tracking how alliances and eligibility evolve over 20 rounds (simulating 2-3 years of real-world cycles).
Key Insights and Ideas
- The Sabotage-Collusion Feedback Loop: Sabotage and collusion aren't just additive; they amplify each other. Sabotage pushes a high-quality paper below the acceptance threshold, freeing up a "seat." Collusion then ensures that seat is filled by an ally’s paper.
- The Scale Paradox: Under perfectly random reviewer assignment, growing the conference size ($A$) protects the system because the chance of meeting an ally ($E = Rm/A$) drops. However, reviewer bidding (where reviewers request specific papers) creates a "floor." In a pool of 10,000, a 20-member ring wins only 0.24% of its papers via random assignment but a staggering 71% when 30% of their bids are granted.
- The Incumbency Loop: Because eligibility to review is often tied to having accepted papers, sabotage becomes a weapon for incumbents. By sabotaging newcomers, incumbents keep them out of the reviewer pool, ensuring that only "established" players (who may be part of existing rings) hold the power to review.
- LLM "Tribalism": The LLM agents exhibited "in-group bias" that went beyond rational payoff optimization. Ring members sabotaged outsiders significantly more than was strictly necessary to help their allies, suggesting that social structures in peer review might breed hostility toward "the out-group."
Key Results
- Signal vs. List: Collusion primarily changes who is accepted (fairness), whereas sabotage erodes the signal (the correlation between scores and actual paper quality).
- The Power of Caps: Limiting the number of "allies" an agent can have to "social brain" layers (like Dunbar’s number of ~15 or ~50) keeps collusion nearly harmless. Without these caps, agents eventually ally with half their research area, causing the "accepted value" (total quality) of the conference to fall by 35%.
- Bidding Risks: The authors provide a design rule for conference chairs: to keep a 20-member ring's success rate below 10%, the "grant rate" for bids must stay below 0.55%.
Limitations and Future Work
- Audit Assumptions: The model assumes a 1% audit rate. In reality, detecting subtle sabotage (giving a 5 instead of a 7) is much harder than detecting blatant collusion.
- Static Quality: The model assumes paper quality is independent of the review system. In reality, if the system is seen as "rigged," high-quality authors might stop submitting altogether.
- Future Directions: The authors suggest future work should focus on "matching quality"—finding the sweet spot where bidding helps find expert reviewers without opening the door to collusion rings. They also aim to calibrate scores against real-world data from platforms like OpenReview.
Summary updated