A Dataset on Malicious Paper Bidding in Peer Review

Steven Jecmen, Minji Yoon, Vincent Conitzer, Nihar B. Shah, Fei Fang

Abstract: Reviewer collusion via bid manipulation poses a significant threat to the integrity of scientific peer review. This paper presents the first human-labeled dataset of malicious paper bidding, generated through a mock conference involving 31 participants. By characterizing common human collusion strategies—such as "negative-in-area" and "cycle" bidding—and benchmarking baseline detection algorithms, the authors provide a critical resource for developing more secure paper assignment systems.

Introduction

In modern computer science conferences, the paper assignment process relies heavily on "bids" submitted by reviewers. These bids indicate a reviewer's interest or expertise in specific submissions, which automated algorithms then use to optimize matches. However, this system is increasingly vulnerable to manipulation. Malicious reviewers may collude in "rings" to ensure they are assigned to a friend's paper to provide a guaranteed positive review, or a rival's paper to "torpedo" it with a negative review.

The primary obstacle in addressing this crisis is the lack of ground-truth data. Real conference bidding data is strictly confidential, and suspected collusion is rarely provable. Consequently, researchers have historically relied on synthetic models of malicious behavior. This paper fills this gap by observing and documenting how humans actually attempt to subvert the system.

Experimental Design and Dataset

The researchers conducted a mock conference at Carnegie Mellon University with 31 graduate students. The experiment was divided into two distinct phases:

  1. Honest Phase: Participants bid on 28 fake paper titles based on their genuine research interests.
  2. Malicious Phase: Participants were organized into collusion rings and tasked with manipulating the assignment algorithm to ensure they were assigned to their group members' papers.

The resulting Malicious Bidding Dataset contains 35 honest and 31 malicious responses mapped across 25 AI subject areas. This dataset provides the first empirical look at the statistical differences between legitimate interest and strategic manipulation.

Comparison of bidding distributions between honest and malicious participants.

As shown in the bidding distributions, malicious actors often exhibit highly specific patterns—such as clustering around a fixed number of positive bids—that deviate from the more varied distributions of honest participants.

Human Collusion Strategies

Through the mock conference, the authors identified several sophisticated strategies employed by human colluders:

  • The "Negative-in-area" Strategy: Reviewers bid negatively on every paper within their expertise area except for their target paper. This forces the assignment algorithm to view the target paper as the only viable match for that reviewer.
  • The "Cycle" Strategy: To evade simple ring detection (where A bids on B and B bids on A), groups bid in a circular fashion (A bids on B, B on C, C on A).
  • Basic Manipulation: Even simple, low-effort strategies were found to be remarkably effective, achieving a 100% success rate in getting reviewers assigned to at least one target paper in the mock setting.

Detection Benchmarks

The authors evaluated three baseline detection methods: Counting Detection (ranking by net negative bids), Ring Detection (identifying reciprocal bidding), and Low-rank Detection (identifying structural deviations).

Effectiveness of detection algorithms across various human-identified collusion strategies.

Key results from the benchmark include:

  • Counting Detection was surprisingly effective, typically placing malicious actors in the top 25% of suspicious candidates.
  • Strategy-Specific Difficulty: While "Popularity"-based manipulation is easily detected, strategies like "Overlap" and "Basic" bidding are significantly harder to isolate, as they blend more effectively into the global bidding structure.
  • Low-rank Failure: Matrix-approximation methods performed poorly, often failing to outperform random guessing, suggesting that malicious bids can be structurally subtle.

Conclusion and Ethical Considerations

While this study provides a vital tool for the research community, the authors acknowledge the limitations of a student-based mock conference compared to high-stakes professional environments. Furthermore, they highlight an inherent "arms race": publishing these datasets may inadvertently teach malicious actors how to better disguise their strategies. Future work must focus on developing "strategy-proof" assignment algorithms that are mathematically resistant to the human behaviors identified in this dataset.

Summary updated

BibTeX
@inproceedings{jecmen2023dataset,
  author = {Steven Jecmen and Minji Yoon and Vincent Conitzer and Nihar B. Shah and Fei Fang},
  title = {{A Dataset on Malicious Paper Bidding in Peer Review}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583424},
}
Figure