TL;DR
This paper analyzes over 10,000 responses to the NLP Reproducibility Checklist, revealing that while the checklist has increased reporting of key experimental details, significant gaps remain, particularly concerning data collection and code availability. The study finds a strong correlation between higher checklist compliance and acceptance rates, but also highlights that papers collecting new data face a significant disadvantage.
What Problem Does This Paper Address?
Reproducibility is the bedrock of scientific progress, yet in rapidly evolving fields like Natural Language Processing (NLP), it's a persistent challenge. The complexity of NLP systems, coupled with the pressure to publish quickly, often leads to insufficient reporting of experimental details. This makes it difficult for other researchers to replicate findings, verify claims, and build upon existing work. To address this, NLP conferences introduced a Reproducibility Checklist, aiming to guide authors in reporting essential information. This paper investigates the effectiveness of this checklist by analyzing a large dataset of author responses.
Methods
The researchers analyzed 10,405 anonymous responses to the NLP Reproducibility Checklist collected from EMNLP 2020, EMNLP 2021, NAACL 2021, and ACL 2021. They correlated checklist responses with paper acceptance status and, for NAACL and ACL 2021, with reviewer-assessed reproducibility scores. Statistical analyses, including regression and correlation, were used to identify trends and relationships between checklist items and publication outcomes. They also performed a principal component analysis (PCA) to understand how different checklist items group together.
Datasets
The study utilized a dataset of 10,405 anonymized responses to the NLP Reproducibility Checklist from four major NLP conferences (EMNLP 2020, EMNLP 2021, NAACL 2021, and ACL 2021). Reviewer scores and feedback were also used for NAACL 2021 and ACL 2021. No new datasets were introduced by this paper.
Key Insights and Ideas
- Checklist Impact: The checklist appears to have successfully increased reporting for several key areas, including efficiency, validation performance, summary statistics, and hyperparameters.
- Acceptance Correlation: Submissions with more "YES" responses to checklist items tend to have higher acceptance rates, suggesting that adherence to reproducibility practices is valued.
- New Data Disadvantage: A significant finding is that papers collecting new data are 5% less likely to be accepted and are perceived as less reproducible, even when they report on data collection. This points to a systemic issue in how new data contributions are valued.
- Code Availability Gap: Less than half of submissions report open-sourcing their code, despite code availability being the strongest predictor of perceived reproducibility.
- Reviewer Perceptions: Reviewers generally found the checklist useful, though a small percentage found it "Not Useful," indicating potential gaps in its coverage or applicability to certain sub-fields.
Key Results
- Increased Reporting: Evidence suggests an increase in reporting for efficiency, validation performance, summary statistics, and hyperparameters after the checklist's introduction.
- Acceptance Rate Boost: Submissions with more "YES" responses to checklist items show a positive correlation with acceptance rates.
- New Data Penalty: Papers collecting new data are 5.1% less likely to be accepted and have a 2% lower reviewer-assessed reproducibility score.
- Code Sharing: Only 46% of submissions claim to open-source their code. However, those that do have an 8% higher reproducibility score, the largest difference observed for any single item.
- Reviewer Usefulness: 59.9% of reviewers found the checklist "Somewhat Useful," and 17.0% found it "Very Useful."
Limitations and Future Work
The authors acknowledge several limitations:
- Self-Reported Data: Checklist responses are self-reported and may not always accurately reflect the paper's content, potentially leading to an overestimation of reporting rates.
- Phasing Differences: Minor variations in checklist item phrasing across conferences could affect direct comparisons.
- Limited Temporal Scope: The analysis covers only four conferences from 2020-2021, limiting the ability to draw long-term temporal trend conclusions.
- Reviewer Data Sparsity: Reviewer-assessed reproducibility data was only available for two of the four conferences.
Future work suggested includes:
- Incentivizing Code Release: Encouraging conferences to incentivize code submission, perhaps through optional deadlines or dedicated tracks.
- Improving Data Collection Reporting: Expanding checklists to cover data collection practices more comprehensively.
- Making Checklist Responses Public: Implementing public access to checklist responses to increase accountability and aid readers.
- Allowing Extended Deadlines: Providing a grace period for submitting code and appendices to facilitate better reporting.
- Investigating Sub-field Specificity: Further exploring why certain checklist items might be less relevant or applicable to specific NLP sub-fields.
Summary updated