This paper, presented at ACL 2022, tackles one of the most frustrating aspects of academic life: the perceived "lottery" of peer review. While most research on paper-reviewer matching focuses on building better recommendation algorithms, Jakobsen and Rogers take a step back to ask the community—authors, reviewers, and Area Chairs (ACs)—what they actually want from the process.
TL;DR
This study surveys the NLP community to reveal a significant disconnect between current automated matching systems and stakeholder needs. The core finding? While "affinity scores" are the industry standard for matching, researchers rank them as the least important factor, instead favoring task-specific expertise and manual bidding.
What Problem Does This Paper Address?
Peer review is the gatekeeper of scientific progress, yet it is often criticized for being arbitrary. Previous experiments (like the famous NeurIPS consistency trials) have shown that reviewer agreement on "good" papers is often close to random.
The authors argue that a primary cause is poor paper-reviewer matching. If a reviewer lacks interest or expertise in a specific sub-field, the resulting review is often shallow, biased, or overly focused on "safe," popular topics. Currently, most conferences treat matching as a black-box optimization problem. This paper seeks to open that box by identifying the specific factors that stakeholders believe lead to high-quality reviews.
Methods
The authors conducted the first comprehensive survey of the NLP community regarding matching preferences. They developed three distinct surveys tailored to the primary roles in the process:
- Authors: Focusing on the quality of feedback received.
- Reviewers: Focusing on the relevance and workload of assigned papers.
- Area Chairs (ACs): Focusing on the tools and information available to make assignments.
To ensure data quality, they employed "trap" questions to filter out inattentive respondents and validated the demographic spread against known conference statistics. They collected 180 valid responses, representing a diverse mix of career stages and research interests.
Datasets
The primary "dataset" is the collection of 180 survey responses. The respondents were primarily academic researchers (62), PhD students (54), and postdocs (32). The authors also categorized the types of research performed by respondents (e.g., engineering experiments, resource papers, position papers) to see if research style influenced matching preferences.
Key Insights and Ideas
- The Affinity Score Paradox: Automated affinity scores (based on embedding similarity between papers and reviewer histories) are the backbone of modern matching systems like OpenReview. However, respondents ranked these as the least important factor for a good match (scoring 1.99/5).
- Task and Method Expertise Rule: The community values "working on the same task" (3.95/5) and "bidding on the paper" (3.85/5) far above automated scores.
- ACs are Flying Blind: Area Chairs reported that their biggest hurdles are clunky interfaces and a lack of structured information about reviewers (such as seniority or specific methodological expertise).
- Research Type Bias: The study found statistically significant differences in preferences based on what people write. For example, authors of reproduction papers care significantly more about the reviewer having worked with the same kind of data, while position paper authors are particularly skeptical of automated affinity scores.
- The "Safe Topic" Trap: Authors admitted to pursuing popular topics just to ensure they get competent reviewers, a trend that risks the "ossification" of the field where non-mainstream research is systematically disadvantaged.
Key Results
The survey highlighted a high rate of dissatisfaction:
- Mismatches: Approximately 30% of reviewers felt their assigned papers were a poor match for their expertise, even when their publication history suggested otherwise.
- Grievances: As shown in Figure 5, authors frequently complained about reviews being "too broad" or reviewers "lacking expertise," while ACs struggled with "insufficient information" to facilitate matches.
- Workflow Preferences: While authors and ACs generally prefer a mix of automated and manual matching, reviewers showed a strong preference for bidding-based systems with manual checks.
Limitations and Future Work
The authors acknowledge that survey participation is voluntary, which may over-represent people who have had negative experiences (a "grievance bias"). Furthermore, while 180 responses is comparable to other major community surveys (like those for ACL Rolling Review), it is still a small slice of the global NLP community.
Future Directions: 1. Interpretable Matching: Instead of just providing a score, matching systems should provide justifications (e.g., "Matched because both work on Low-Resource MT using Transformers"). 2. Better AC Tooling: Providing ACs with structured data on reviewer seniority and specific "tags" (like the ACM CCS concepts) rather than just raw publication links. 3. Exploratory Matching: Implementing a combination of bidding-based matches (for interest) and random matches within a sub-field (to encourage diversity and prevent collusion).
Summary updated