What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Terne Sasha Thorn Jakobsen, Anna Rogers

Introduction

Peer review serves as the primary gatekeeper of scientific progress, yet the process is increasingly criticized for being arbitrary and inconsistent. High-profile experiments at major venues like NeurIPS have demonstrated that reviewer agreement on paper quality is often little better than random chance. While many researchers have attempted to solve this "review lottery" by developing more sophisticated recommendation algorithms, Jakobsen and Rogers take a different approach. They investigate the underlying criteria of the matching process itself, asking the community: What actually makes a good match?

This study provides a data-driven look into the "black box" of conference peer review within the Natural Language Processing (NLP) community. By surveying the values of authors, reviewers, and Area Chairs (ACs), the authors reveal a significant disconnect between the automated systems currently in use and the preferences of the researchers they serve.

Methodology

To capture a holistic view of the field, the researchers conducted a comprehensive survey in late 2021, targeting three distinct stakeholder roles: Area Chairs, reviewers, and authors. The study gathered 180 validated responses (30 ACs, 77 reviewers, and 73 authors), primarily from academic researchers, PhD students, and postdocs.

To ensure data integrity, the authors employed "satisficing theory," incorporating "trap" questions to filter out inattentive respondents and verifying the logical consistency of reported roles and timelines. The analysis combined quantitative measures—using 5-point Likert scales and Fisher’s Exact tests for statistical significance—with qualitative analysis of open-ended comments. Unlike traditional studies that propose a new algorithm, this "community-driven" research identifies the specific criteria that future algorithms should prioritize.

The "Affinity Score" Paradox

The most striking finding of the study is the rejection of current industry standards. Most major conferences rely heavily on automated "affinity scores," which are generated by comparing vector embeddings of submitted papers against a reviewer’s publication history. However, survey respondents ranked these scores as the least important factor in a good match, giving them an average importance score of only 1.99 out of 5.

In contrast, the community expressed a strong preference for matches based on specific technical alignment:

  1. Task Expertise (3.95/5): The reviewer has worked on the same specific task (e.g., summarization or parsing).
  2. Manual Bidding (3.85/5): The reviewer explicitly expressed interest in the paper.
  3. Methodological Expertise (3.72/5): The reviewer has worked with the same underlying methods (e.g., reinforcement learning or Bayesian inference).

This suggests that while automated systems look for general subject-area overlap, researchers value deep, specialized knowledge of the specific problems and techniques discussed in a paper.

Systemic Friction and Avoidable Mismatches

The survey also highlighted significant operational failures in the current peer-review ecosystem. Approximately 30% of reported mismatches were classified as "avoidable," meaning the reviewer’s public publication history clearly indicated they were not a fit for the assigned paper, yet the automated system assigned them regardless.

Other key indicators of systemic friction include:

  • Dissatisfaction among Area Chairs: Only 6.7% of ACs reported having "no issues" with paper matching in their most recent experience.
  • Transparency Gap: Only 23% of authors and reviewers understood the process used to match papers at their last venue.
  • Workload Imbalance: There is a consistent gap between "optimal" and "actual" workloads. Reviewers reported a preference for roughly 2.8 papers per cycle but received an average of 3.3, leading to shallower reviews.

Discussion: Risks and Recommendations

While manual bidding is highly valued by the community, the authors caution that relying solely on it can lead to "field ossification." If reviewers only bid on popular or "safe" topics, innovative, interdisciplinary, or non-mainstream research may be systematically ignored. To mitigate this, the authors suggest a hybrid approach: combining manual bidding with a degree of randomized assignment within a relevant matching area to ensure diversity.

Furthermore, the study suggests that preferences are not monolithic. For instance, researchers who write "position papers" are significantly less likely to trust automated affinity scores than those focused on engineering experiments, highlighting the need for matching systems that can handle different types of research contributions.

Conclusion

The findings of Jakobsen and Rogers call for a shift toward interpretable paper-reviewer matching. Rather than relying on opaque numerical scores, future conference management systems should provide rationales for assignments (e.g., "This reviewer was assigned because they have published three papers on this specific task"). By aligning matching algorithms with community values—prioritizing task-specific expertise and increasing transparency—the field can move toward a more equitable and rigorous peer-review process.

Summary updated

Abstract

Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.
BibTeX
@inproceedings{jakobsen2022what,
  author = {Terne Sasha Thorn Jakobsen and Anna Rogers},
  title = {{What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.354},
}
Figure