TL;DR
This report details the ACL 2023 program chairs' extensive efforts to improve the conference's peer review process, focusing on review quality, reviewer-paper matching, and participant incentives. It offers valuable insights into the challenges and successes of running a large-scale NLP conference, providing actionable recommendations for future organizers.
What Problem Does This Paper Address?
The paper addresses the perennial challenges of large-scale academic conference peer review, particularly within the rapidly growing field of Natural Language Processing (NLP). As submission numbers increase, ensuring high-quality reviews, fair reviewer-paper matching, and efficient decision-making becomes increasingly difficult. The authors aim to improve transparency, fairness, and overall effectiveness of the peer review process, which is crucial for maintaining the quality and integrity of scientific publications. They also highlight issues like reviewer fatigue, potential biases, and the need for better incentives for all participants.
Methods
The authors implemented several novel approaches and analyzed existing data to improve the ACL 2023 peer review process. Key initiatives included:
- Improved Reviewer-Paper Matching: A new "Area-Contribution-Language" (ACL) matching system was developed, which considers topical sub-areas, contribution types (e.g., NLP engineering experiment, theory), and target language. This aimed to move beyond simple keyword matching and improve reviewer engagement.
- Enhanced Decision Support for Chairs: Updated guidelines for Senior Area Chairs (SACs) and Area Chairs (ACs) were created, and a system for providing "match rationales" was introduced to help chairs understand why a reviewer was assigned to a paper.
- Soundness/Excitement Scores: The traditional "Overall Recommendation" score was replaced with two separate scores: "Soundness" (technical merit) and "Excitement" (novelty/impact). This was intended to differentiate between solid but incremental work and groundbreaking research, and to focus author responses on objective issues.
- Review Issue Flagging: Authors were given a mechanism to flag specific issues with reviews, allowing ACs to address problematic feedback more systematically.
- Responsible NLP Checklist: The checklist, previously used internally, was made a mandatory part of all published papers to encourage better reporting on reproducibility, data collection, and broader impacts.
- Reviewer Awards: An initiative to recognize outstanding reviewers and chairs was implemented, combining reputational and material incentives (vouchers for conference attendance).
Datasets
The paper primarily analyzes internal data from ACL 2023, including:
- Submission data (4559 direct, 305 from ARR).
- Reviewer data (4490 reviewers).
- Author data (13,658 authors).
- Exit survey responses from Senior Area Chairs (SACs), Area Chairs (ACs), reviewers, and authors.
- Data from START (the conference management system).
- Semantic Scholar profiles for reviewer-paper matching.
No new datasets were introduced in this paper; instead, it focuses on analyzing the data generated by the ACL 2023 peer review process itself.
Key Insights and Ideas
- The "Area-Contribution-Language" matching system shows promise: While not perfect, this multi-dimensional matching approach significantly improved reviewer activity and ensured better matches for non-English language papers and diverse contribution types.
- Soundness and Excitement scores are a step in the right direction: The new scoring system received generally positive feedback and showed a clearer correlation with acceptance outcomes than previous methods, helping to differentiate between solid and groundbreaking research.
- Author flagging of review issues is valuable: This feature was overwhelmingly supported by authors and provided crucial feedback for identifying problematic reviews, though ACs need further training to address these issues effectively.
- Preprints and perceived author familiarity can introduce bias: Papers with preprints or where reviewers thought they knew the authors received higher scores, suggesting a need for careful consideration of anonymity policies.
- Reviewer agreement is still a challenge, but Soundness is better than Excitement: While inter-reviewer agreement remains a concern, the Soundness score showed higher agreement than the more subjective Excitement score, indicating that more concrete criteria can improve consistency.
- Incentives matter: The Responsible NLP Checklist and reviewer awards demonstrated positive impacts on reporting practices and reviewer engagement.
Key Results
- Acceptance Rates: ARR submissions had a higher acceptance rate than direct submissions.
- Reviewer Activity: Contribution type matching significantly increased reviewer activity by 1.178 times.
- Reviewer Scores: Soundness and Excitement scores were strong predictors of acceptance, with Soundness having a more pronounced effect.
- Review Issue Flagging: 12.9% of reviews were flagged by authors, with the most common issues related to reviewer heuristics (e.g., "not novel," "too simple").
- Preprints: Papers with preprints received higher scores and were more likely to be nominated for awards.
- Reviewer Agreement: Krippendorff's alpha for Soundness was around 0.3, while for Excitement it was lower, indicating moderate agreement.
Limitations and Future Work
- ARR Hybrid Process: The paper acknowledges potential unfairness in the hybrid ARR/direct submission process, with some SACs suggesting separate acceptance rates or processes.
- Reviewer Matching Nuances: While ACL matching improved things, there's still room for refinement, particularly in keyword definitions and reviewer recruitment for underrepresented languages.
- Reviewer Discussion Engagement: While ACs initiated discussions, reviewer engagement in these forums was not always high.
- Reviewer Confidence vs. Experience: Reviewer confidence scores did not strongly correlate with actual experience or the number of reported review issues.
- Anonymity and Bias: The impact of preprints and perceived author familiarity on review scores suggests a need for further research into causal mechanisms and policy adjustments.
- Reviewer Assignment Interface: The current START platform lacks a convenient interface for ACs to check assignments and explore alternatives, leading to manual adjustments and potential delays.
- Post-Acceptance Decision Litigation: The paper notes a surprising volume of author appeals, suggesting a need for a clearer process for handling such cases or a firm stance on finality.
- Workload for Program Chairs: The authors recommend expanding the volunteer roles and creating dedicated positions for peer review, best paper selection, and scheduling to manage the increasing workload.
- Ethics Review Process: While the ethics review was useful, it faced challenges with conditional accepts and required careful coordination.
Summary updated