Abstract: Reviewer collusion via bid manipulation poses a significant threat to the integrity of scientific peer review. This paper presents the first human-labeled dataset of malicious paper bidding, generated through a mock conference involving 31 participants. By characterizing common human collusion strategies—such as "negative-in-area" and "cycle" bidding—and benchmarking baseline detection algorithms, the authors provide a critical resource for developing more secure paper assignment systems.
Introduction
In modern computer science conferences, the paper assignment process relies heavily on "bids" submitted by reviewers. These bids indicate a reviewer's interest or expertise in specific submissions, which automated algorithms then use to optimize matches. However, this system is increasingly vulnerable to manipulation. Malicious reviewers may collude in "rings" to ensure they are assigned to a friend's paper to provide a guaranteed positive review, or a rival's paper to "torpedo" it with a negative review.
The primary obstacle in addressing this crisis is the lack of ground-truth data. Real conference bidding data is strictly confidential, and suspected collusion is rarely provable. Consequently, researchers have historically relied on synthetic models of malicious behavior. This paper fills this gap by observing and documenting how humans actually attempt to subvert the system.
Experimental Design and Dataset
The researchers conducted a mock conference at Carnegie Mellon University with 31 graduate students. The experiment was divided into two distinct phases:
- Honest Phase: Participants bid on 28 fake paper titles based on their genuine research interests.
- Malicious Phase: Participants were organized into collusion rings and tasked with manipulating the assignment algorithm to ensure they were assigned to their group members' papers.
The resulting Malicious Bidding Dataset contains 35 honest and 31 malicious responses mapped across 25 AI subject areas. This dataset provides the first empirical look at the statistical differences between legitimate interest and strategic manipulation.
As shown in the bidding distributions, malicious actors often exhibit highly specific patterns—such as clustering around a fixed number of positive bids—that deviate from the more varied distributions of honest participants.
Human Collusion Strategies
Through the mock conference, the authors identified several sophisticated strategies employed by human colluders:
- The "Negative-in-area" Strategy: Reviewers bid negatively on every paper within their expertise area except for their target paper. This forces the assignment algorithm to view the target paper as the only viable match for that reviewer.
- The "Cycle" Strategy: To evade simple ring detection (where A bids on B and B bids on A), groups bid in a circular fashion (A bids on B, B on C, C on A).
- Basic Manipulation: Even simple, low-effort strategies were found to be remarkably effective, achieving a 100% success rate in getting reviewers assigned to at least one target paper in the mock setting.
Detection Benchmarks
The authors evaluated three baseline detection methods: Counting Detection (ranking by net negative bids), Ring Detection (identifying reciprocal bidding), and Low-rank Detection (identifying structural deviations).
Key results from the benchmark include:
- Counting Detection was surprisingly effective, typically placing malicious actors in the top 25% of suspicious candidates.
- Strategy-Specific Difficulty: While "Popularity"-based manipulation is easily detected, strategies like "Overlap" and "Basic" bidding are significantly harder to isolate, as they blend more effectively into the global bidding structure.
- Low-rank Failure: Matrix-approximation methods performed poorly, often failing to outperform random guessing, suggesting that malicious bids can be structurally subtle.
Conclusion and Ethical Considerations
While this study provides a vital tool for the research community, the authors acknowledge the limitations of a student-based mock conference compared to high-stakes professional environments. Furthermore, they highlight an inherent "arms race": publishing these datasets may inadvertently teach malicious actors how to better disguise their strategies. Future work must focus on developing "strategy-proof" assignment algorithms that are mathematically resistant to the human behaviors identified in this dataset.
Summary updated