Abstract
Scientific progress in Natural Language Processing (NLP) relies on the ability to replicate experimental results. To address the growing complexity of models and the resulting "under-reporting" of critical details, major *CL conferences introduced a mandatory Reproducibility Checklist in 2020. This paper presents a meta-analysis of 10,405 anonymized checklist responses from EMNLP, NAACL, and ACL. Our findings indicate that while the checklist has improved reporting standards, there is a significant "data gap" for new datasets and evidence of "bad faith" reporting. We also find that code availability remains the strongest predictor of perceived reproducibility.
Introduction
As NLP systems become more sophisticated, the pressure to publish frequently often leads researchers to omit essential details such as hyperparameters, hardware configurations, and data processing steps. This lack of transparency hinders the community's ability to build upon previous work. In response, the NLP community implemented a mandatory Reproducibility Checklist. This study evaluates the effectiveness of this administrative requirement: Does it change reporting behavior? Does a "reproducible" paper have a higher chance of acceptance?
Methodology
The researchers analyzed 10,405 anonymized responses from four major conferences: EMNLP 2020, EMNLP 2021, NAACL 2021, and ACL 2021. The study employed several analytical techniques:
- Correlation Analysis: Linking checklist responses to acceptance rates and reviewer-assigned reproducibility scores.
- Diachronic Comparison: Benchmarking current self-reported rates against a manual audit of 50 papers from EMNLP 2018.
- Principal Component Analysis (PCA): Identifying clusters of reporting behavior, such as the relationship between open-sourcing code and documenting data.
Key Findings
The "All-Yes" Red Flag and Acceptance Trends
A primary finding is the positive correlation between the number of "YES" responses on the checklist and a paper's acceptance rate. However, this trend is not monotonic. Papers that claim "YES" for every single item often experience a drop in acceptance rates. This suggests a "bad faith" reporting phenomenon where authors may check boxes to satisfy requirements without actually providing the necessary information in the manuscript.
The Data Collection Gap
A significant "data gap" exists in the current research landscape. Submissions introducing new datasets are 5% less likely to be accepted than those utilizing existing benchmarks. Furthermore, these papers receive reproducibility scores that are 2% lower on average. This indicates that the community lacks established norms for documenting new data collection and processing, or that reviewers penalize the inherent complexity of new data work.
Code Availability and Reviewer Perception
Providing a link to source code is the single most influential factor in a reviewer's perception of reproducibility, increasing scores by an average of 0.18 points. Despite this, only about 46% of submissions claim to have open-sourced their code at the time of submission. Many authors cite "crunch time" as a reason for omitting code or data, suggesting these tasks are often deprioritized as deadlines approach.
Reviewer Sentiment and Impact
The checklist is generally viewed positively by the community, with 77% of reviewers finding it useful for evaluating submissions. However, the impact of reproducibility scores on final acceptance is secondary to general recommendation scores.
As shown in the figure above, while higher reproducibility scores (AvgReprod) correlate with higher acceptance, the general recommendation score (AvgRec) is a much steeper and more definitive predictor of a paper's success.
Conclusion and Future Directions
The Reproducibility Checklist has successfully increased the reporting of efficiency metrics and hyperparameters compared to pre-2020 standards. However, self-reporting bias remains a challenge. To further improve the field, the authors propose:
- Staggered Deadlines: Allowing a one-week grace period after the main paper deadline for submitting code and appendices.
- Specialized Data Checklists: Implementing specific prompts for data collection practices to bridge the "data gap."
- Public Accountability: Making checklist responses public for accepted papers to ensure transparency.
Summary updated