Abstract
Assigning reviewers to scientific papers is a critical yet logistically complex task for large-scale conferences. While automated algorithms facilitate this process using reviewer bids and text similarity, evaluating new assignment policies typically requires expensive randomized controlled trials. This paper presents a framework for the counterfactual evaluation of peer-review assignment policies using existing data. By leveraging the randomness introduced by anti-collusion algorithms, the authors employ off-policy evaluation (OPE) and novel partial identification techniques to estimate the performance of alternative policies. Results from TPDP ’21 and AAAI ’22 suggest that prioritizing automated text similarity over reviewer bids can improve review quality with minimal costs to the conference's overall matching score.
Introduction
The integrity of the scientific process relies heavily on the quality of peer review. In major venues such as AAAI or NeurIPS, thousands of papers must be matched to thousands of reviewers. To manage this, organizers use automated algorithms that weigh three primary signals: reviewer bids, text similarity (matching paper content to a reviewer’s publication history), and subject area expertise.
However, optimizing these weights is challenging. Conference organizers often lack the means to predict how a change in policy—such as increasing the weight of text similarity—will affect the quality of the resulting reviews. Traditionally, such questions required A/B testing, which is prohibitively expensive in the context of peer review due to the limited pool of volunteer reviewers and the risk of compromising the review process for a subset of papers. This work addresses the need for a "counterfactual" approach: evaluating "what if" scenarios using historical data.
Methodology
Exploiting Security Randomness
The authors utilize Off-Policy Evaluation (OPE), a method common in recommendation systems, to estimate the outcomes of new policies. The primary challenge in OPE is the requirement for randomness in the original data to ensure that various reviewer-paper pairings were at least possible. Interestingly, the authors find this necessary randomness in the anti-collusion measures recently adopted by major conferences. These algorithms introduce intentional stochasticity to prevent reviewers from strategically matching with papers from colleagues or friends. This security-driven randomness provides the statistical overlap required to simulate alternative assignment universes.
Partial Identification and Bounding
A significant hurdle in OPE is the "positivity violation," which occurs when a proposed policy wants to assign a pair that had zero probability of being assigned in the original data. To address this, the authors move beyond simple point estimates and instead calculate bounds (minimum and maximum possible values) for review quality. They introduce two key assumptions to tighten these bounds:
- Monotonicity: This assumes that if a reviewer has higher bids and higher text similarity for Paper A than for Paper B, their review quality for Paper A should not be lower than for Paper B.
- Lipschitz Smoothness: This assumes that reviewers with similar profiles (in terms of bids and similarity scores) should produce reviews of comparable quality.
By framing the bounding problem as a two-level linear programming optimization, the authors can provide conservative yet useful estimates of how a new policy would perform.
Empirical Evaluation
The framework was tested using data from two distinct venues:
- TPDP ’21: A workshop on Differential Privacy (95 papers, 35 reviewers).
- AAAI ’22: A major AI conference (8,450 papers, 3,145 reviewers).
Quality was measured using reviewers' self-reported expertise and confidence scores. The researchers compared the predicted expertise under various weighting schemes for bids and text similarity.
Key Insights
The Efficacy of Text Similarity
In both datasets, the models indicated that increasing the weight of automated text similarity ($w_{text}$) relative to reviewer bids led to higher predicted expertise. For example, in the TPDP data, shifting the weight from 0.75 to 0.9 resulted in a significant improvement in average expertise. This suggests that reviewers may not always be the most accurate judges of their own suitability, or that strategic bidding may occasionally diverge from actual expertise.
The Low Cost of Randomization
Organizers often worry that introducing randomness for security purposes will degrade the quality of the conference. However, this study finds that while randomization slightly reduces the mathematical similarity scores of matches, the actual reported expertise of reviewers drops only marginally. This supports the continued use of randomized assignments as a robust security measure that does not significantly compromise academic standards.
Robustness of Bounding Techniques
The authors demonstrated that their Monotonicity and Lipschitz-based bounds were significantly tighter than standard Manski bounds, which make no assumptions about the underlying data. This proves that incorporating common-sense domain knowledge into statistical models can yield much more actionable policy insights.
Limitations and Future Work
While the study provides a powerful tool for organizers, it operates under the Stable Unit Treatment Value Assumption (SUTVA), assuming that the quality of one review is independent of a reviewer's other assignments. In practice, "batch effects" or reviewer fatigue could influence quality. Furthermore, while text similarity may improve expertise scores, it is important to consider reviewer welfare; forcing reviewers to evaluate papers they did not bid on might lead to burnout or dissatisfaction. Future research should explore the multi-objective optimization of expertise, security, and reviewer satisfaction.
Conclusion
This work demonstrates that the data already collected by large conferences—specifically the randomness introduced for security—can be repurposed to rigorously evaluate and improve assignment policies. By moving toward a counterfactual evaluation framework, conference organizers can make data-driven decisions to enhance the quality and integrity of the peer-review process without the need for costly and risky experiments.
Summary updated