Counterfactual Evaluation of Peer-Review Assignment Policies

Martin Saveski, Steven Jecmen, Nihar B Shah, Johan Ugander

Abstract

Assigning reviewers to scientific papers is a critical yet logistically complex task for large-scale conferences. While automated algorithms facilitate this process using reviewer bids and text similarity, evaluating new assignment policies typically requires expensive randomized controlled trials. This paper presents a framework for the counterfactual evaluation of peer-review assignment policies using existing data. By leveraging the randomness introduced by anti-collusion algorithms, the authors employ off-policy evaluation (OPE) and novel partial identification techniques to estimate the performance of alternative policies. Results from TPDP ’21 and AAAI ’22 suggest that prioritizing automated text similarity over reviewer bids can improve review quality with minimal costs to the conference's overall matching score.

Introduction

The integrity of the scientific process relies heavily on the quality of peer review. In major venues such as AAAI or NeurIPS, thousands of papers must be matched to thousands of reviewers. To manage this, organizers use automated algorithms that weigh three primary signals: reviewer bids, text similarity (matching paper content to a reviewer’s publication history), and subject area expertise.

However, optimizing these weights is challenging. Conference organizers often lack the means to predict how a change in policy—such as increasing the weight of text similarity—will affect the quality of the resulting reviews. Traditionally, such questions required A/B testing, which is prohibitively expensive in the context of peer review due to the limited pool of volunteer reviewers and the risk of compromising the review process for a subset of papers. This work addresses the need for a "counterfactual" approach: evaluating "what if" scenarios using historical data.

Methodology

Exploiting Security Randomness

The authors utilize Off-Policy Evaluation (OPE), a method common in recommendation systems, to estimate the outcomes of new policies. The primary challenge in OPE is the requirement for randomness in the original data to ensure that various reviewer-paper pairings were at least possible. Interestingly, the authors find this necessary randomness in the anti-collusion measures recently adopted by major conferences. These algorithms introduce intentional stochasticity to prevent reviewers from strategically matching with papers from colleagues or friends. This security-driven randomness provides the statistical overlap required to simulate alternative assignment universes.

Partial Identification and Bounding

A significant hurdle in OPE is the "positivity violation," which occurs when a proposed policy wants to assign a pair that had zero probability of being assigned in the original data. To address this, the authors move beyond simple point estimates and instead calculate bounds (minimum and maximum possible values) for review quality. They introduce two key assumptions to tighten these bounds:

  1. Monotonicity: This assumes that if a reviewer has higher bids and higher text similarity for Paper A than for Paper B, their review quality for Paper A should not be lower than for Paper B.
  2. Lipschitz Smoothness: This assumes that reviewers with similar profiles (in terms of bids and similarity scores) should produce reviews of comparable quality.

By framing the bounding problem as a two-level linear programming optimization, the authors can provide conservative yet useful estimates of how a new policy would perform.

Empirical Evaluation

The framework was tested using data from two distinct venues:

  • TPDP ’21: A workshop on Differential Privacy (95 papers, 35 reviewers).
  • AAAI ’22: A major AI conference (8,450 papers, 3,145 reviewers).

Quality was measured using reviewers' self-reported expertise and confidence scores. The researchers compared the predicted expertise under various weighting schemes for bids and text similarity.

Key Insights

The Efficacy of Text Similarity

In both datasets, the models indicated that increasing the weight of automated text similarity ($w_{text}$) relative to reviewer bids led to higher predicted expertise. For example, in the TPDP data, shifting the weight from 0.75 to 0.9 resulted in a significant improvement in average expertise. This suggests that reviewers may not always be the most accurate judges of their own suitability, or that strategic bidding may occasionally diverge from actual expertise.

The Low Cost of Randomization

Organizers often worry that introducing randomness for security purposes will degrade the quality of the conference. However, this study finds that while randomization slightly reduces the mathematical similarity scores of matches, the actual reported expertise of reviewers drops only marginally. This supports the continued use of randomized assignments as a robust security measure that does not significantly compromise academic standards.

Robustness of Bounding Techniques

The authors demonstrated that their Monotonicity and Lipschitz-based bounds were significantly tighter than standard Manski bounds, which make no assumptions about the underlying data. This proves that incorporating common-sense domain knowledge into statistical models can yield much more actionable policy insights.

Limitations and Future Work

While the study provides a powerful tool for organizers, it operates under the Stable Unit Treatment Value Assumption (SUTVA), assuming that the quality of one review is independent of a reviewer's other assignments. In practice, "batch effects" or reviewer fatigue could influence quality. Furthermore, while text similarity may improve expertise scores, it is important to consider reviewer welfare; forcing reviewers to evaluate papers they did not bid on might lead to burnout or dissatisfaction. Future research should explore the multi-objective optimization of expertise, security, and reviewer satisfaction.

Conclusion

This work demonstrates that the data already collected by large conferences—specifically the randomness introduced for security—can be repurposed to rigorously evaluate and improve assignment policies. By moving toward a counterfactual evaluation framework, conference organizers can make data-driven decisions to enhance the quality and integrity of the peer-review process without the need for costly and risky experiments.

Summary updated

Abstract

Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working to maximize the quality of the resulting reviews. A key challenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map to changes in review quality. In this work, we leverage recently proposed policies that introduce randomness in peer-review assignment—in order to mitigate fraud—as a valuable opportunity to evaluate counterfactual assignment policies. Specifically, we exploit how such randomized assignments provide a positive probability of observing the reviews of many assignment policies of interest. To address challenges in applying standard off-policy evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and Lipschitz smoothness assumptions for the mapping between reviewer-paper covariates and outcomes. We apply our methods to peer-review data from two computer science venues: the TPDP'21 workshop (95 papers and 35 reviewers) and the AAAI'22 conference (8,450 papers and 3,145 reviewers). We consider estimates of (i) the effect on review quality when changing weights in the assignment algorithm, e.g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and (ii) the "cost of randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match. We find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces the review quality. Our methods for partial identification may be of independent interest, while our off-policy approach can likely find use in evaluating a broad class of algorithmic matching systems.
BibTeX
@inproceedings{saveski2023counterfactual,
  author = {Martin Saveski and Steven Jecmen and Nihar B Shah and Johan Ugander},
  title = {{Counterfactual Evaluation of Peer-Review Assignment Policies}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=rhIfzCZoXG},
}
Figure