Introduction
As academic conferences grow in scale, the process of assigning papers to reviewers has become increasingly complex. In major venues like AAAI or NeurIPS, thousands of reviewers must navigate thousands of submissions. Current "blind" bidding systems often fail to coordinate this effort effectively, leading to skewed demand where popular papers are over-subscribed while "orphan" papers receive no bids at all. This paper proposes a market-inspired "trading post" bidding scheme designed to align individual reviewer incentives with the global needs of the conference.
The Inefficiencies of Blind Bidding
Traditional bidding systems typically require reviewers to select a fixed number of papers (e.g., 75) without any information regarding the current demand for those papers. This lack of transparency creates two primary issues:
- Skewed Demand: A small subset of "hot" papers attracts the vast majority of bids, leaving many papers without qualified reviewers. Program chairs must then manually assign these orphans, often to reviewers with little interest or expertise in the topic.
- Wasted Reviewer Effort: Reviewers spend significant time evaluating and bidding on papers they have a statistically negligible chance of receiving due to high competition.
The Trading Post Bidding Scheme
The proposed solution treats paper assignment as a market coordination problem. By introducing dynamic "prices" that reflect demand, the system guides reviewers toward under-subscribed papers.
Dynamic "iPrices"
Each paper is assigned an "inverse price" ($p_j$) based on the ratio of required reviewers ($r$) to current bidders ($d_j$). For example, if a paper needs 3 reviewers and has 6 bidders, its iprice is 0.5. If it has only 1 bidder, its iprice is 1.0. High demand leads to a lower iprice, effectively making the paper "more expensive" to bid on.
Budget-Based Bidding
Instead of a fixed quota of papers, reviewers are given a total budget ($R$). Bidding on a paper consumes a portion of this budget equal to its current iprice. This mechanism incentivizes reviewers to seek out "cheaper" (under-demanded) papers where their bid carries more weight, increasing their probability of a successful match.
Experimental Evaluation
The researchers validated the scheme using real-world bidding data from PrefLib (DP1-DP5), major AI conferences (DA1-DA3), and ICLR-18. They simulated reviewer behavior using different strategies, including a "Greedy" model where reviewers are price-sensitive and prioritize papers with higher iprices.
Key Results and Insights
The simulations revealed that the trading post scheme significantly outperforms traditional methods across several metrics:
- Elimination of Orphans: By revealing demand, the system naturally pushes reviewers toward orphan papers. The number of papers receiving zero bids dropped substantially, leading to higher overall system satisfaction.
- Superior Matching Quality: Reviewers achieved better matches with papers that aligned with their expertise. Interestingly, the "Greedy" strategy—where reviewers actively avoid over-subscribed papers—resulted in socially superior outcomes for the entire system.
- Reduced Reviewer Burden: The market mechanism achieved these improvements with fewer total positive bids per reviewer. Reviewers no longer need to hit arbitrary quotas; they simply need to allocate their budget effectively.
Conclusion
By introducing transparency and market-based incentives into the bidding process, the Trading Post scheme transforms a chaotic coordination problem into an efficient allocation mechanism. The results suggest that providing reviewers with simple demand signals not only reduces their workload but also ensures a more equitable and high-quality assignment for the entire conference community.
Summary updated