A Market-Inspired Bidding Scheme for Peer Review Paper Assignment

Reshef Meir, Jerome Lang, Julien Lesca, Nicholas Mattei, Natan KaminskyAAAI 2021 · OpenReview

Introduction

As academic conferences grow in scale, the process of assigning papers to reviewers has become increasingly complex. In major venues like AAAI or NeurIPS, thousands of reviewers must navigate thousands of submissions. Current "blind" bidding systems often fail to coordinate this effort effectively, leading to skewed demand where popular papers are over-subscribed while "orphan" papers receive no bids at all. This paper proposes a market-inspired "trading post" bidding scheme designed to align individual reviewer incentives with the global needs of the conference.

The Inefficiencies of Blind Bidding

Traditional bidding systems typically require reviewers to select a fixed number of papers (e.g., 75) without any information regarding the current demand for those papers. This lack of transparency creates two primary issues:

  1. Skewed Demand: A small subset of "hot" papers attracts the vast majority of bids, leaving many papers without qualified reviewers. Program chairs must then manually assign these orphans, often to reviewers with little interest or expertise in the topic.
  2. Wasted Reviewer Effort: Reviewers spend significant time evaluating and bidding on papers they have a statistically negligible chance of receiving due to high competition.

The Trading Post Bidding Scheme

The proposed solution treats paper assignment as a market coordination problem. By introducing dynamic "prices" that reflect demand, the system guides reviewers toward under-subscribed papers.

Dynamic "iPrices"

Each paper is assigned an "inverse price" ($p_j$) based on the ratio of required reviewers ($r$) to current bidders ($d_j$). For example, if a paper needs 3 reviewers and has 6 bidders, its iprice is 0.5. If it has only 1 bidder, its iprice is 1.0. High demand leads to a lower iprice, effectively making the paper "more expensive" to bid on.

Budget-Based Bidding

Instead of a fixed quota of papers, reviewers are given a total budget ($R$). Bidding on a paper consumes a portion of this budget equal to its current iprice. This mechanism incentivizes reviewers to seek out "cheaper" (under-demanded) papers where their bid carries more weight, increasing their probability of a successful match.

Experimental Evaluation

The researchers validated the scheme using real-world bidding data from PrefLib (DP1-DP5), major AI conferences (DA1-DA3), and ICLR-18. They simulated reviewer behavior using different strategies, including a "Greedy" model where reviewers are price-sensitive and prioritize papers with higher iprices.

System and bidder satisfaction comparison for DA2 and DP4 datasets under the proposed bidding scheme.

Key Results and Insights

The simulations revealed that the trading post scheme significantly outperforms traditional methods across several metrics:

  • Elimination of Orphans: By revealing demand, the system naturally pushes reviewers toward orphan papers. The number of papers receiving zero bids dropped substantially, leading to higher overall system satisfaction.
  • Superior Matching Quality: Reviewers achieved better matches with papers that aligned with their expertise. Interestingly, the "Greedy" strategy—where reviewers actively avoid over-subscribed papers—resulted in socially superior outcomes for the entire system.
  • Reduced Reviewer Burden: The market mechanism achieved these improvements with fewer total positive bids per reviewer. Reviewers no longer need to hit arbitrary quotas; they simply need to allocate their budget effectively.
Performance of utilitarian assignments across different bidding strategies, showing the advantage of the market-inspired approach.

Conclusion

By introducing transparency and market-based incentives into the bidding process, the Trading Post scheme transforms a chaotic coordination problem into an efficient allocation mechanism. The results suggest that providing reviewers with simple demand signals not only reduces their workload but also ensures a more equitable and high-quality assignment for the entire conference community.

Summary updated

Abstract

We propose a market-inspired bidding scheme for the assignment of paper reviews in large academic conferences. We provide an analysis of the incentives of reviewers during the bidding phase, when reviewers have both private costs and some information about the demand for each paper; and their goal is to obtain the best possible k papers for a predetermined k. We show that by assigning `budgets' to reviewers and a `price' for every paper that is (roughly) proportional to its demand, the best response of a reviewer is to bid sincerely, i.e., on her most favorite papers, and match the budget even when it is not enforced. This game-theoretic analysis is based on a simple, prototypical assignment algorithm. We show via extensive simulations on bidding data from real conferences, that our bidding scheme would substantially improve both the bid distribution and the resulting assignment.
BibTeX
@inproceedings{meir2021market,
  author = {Reshef Meir and Jerome Lang and Julien Lesca and Nicholas Mattei and Natan Kaminsky},
  title = {{A Market-Inspired Bidding Scheme for Peer Review Paper Assignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=sxh3WiUf9v},
}
Figure