Reproducibility in NLP: What Have We Learned from the Checklist?

Ian Magnusson, Noah A. Smith, Jesse Dodge

Abstract

Scientific progress in Natural Language Processing (NLP) relies on the ability to replicate experimental results. To address the growing complexity of models and the resulting "under-reporting" of critical details, major *CL conferences introduced a mandatory Reproducibility Checklist in 2020. This paper presents a meta-analysis of 10,405 anonymized checklist responses from EMNLP, NAACL, and ACL. Our findings indicate that while the checklist has improved reporting standards, there is a significant "data gap" for new datasets and evidence of "bad faith" reporting. We also find that code availability remains the strongest predictor of perceived reproducibility.

Introduction

As NLP systems become more sophisticated, the pressure to publish frequently often leads researchers to omit essential details such as hyperparameters, hardware configurations, and data processing steps. This lack of transparency hinders the community's ability to build upon previous work. In response, the NLP community implemented a mandatory Reproducibility Checklist. This study evaluates the effectiveness of this administrative requirement: Does it change reporting behavior? Does a "reproducible" paper have a higher chance of acceptance?

Methodology

The researchers analyzed 10,405 anonymized responses from four major conferences: EMNLP 2020, EMNLP 2021, NAACL 2021, and ACL 2021. The study employed several analytical techniques:

  • Correlation Analysis: Linking checklist responses to acceptance rates and reviewer-assigned reproducibility scores.
  • Diachronic Comparison: Benchmarking current self-reported rates against a manual audit of 50 papers from EMNLP 2018.
  • Principal Component Analysis (PCA): Identifying clusters of reporting behavior, such as the relationship between open-sourcing code and documenting data.

Key Findings

A primary finding is the positive correlation between the number of "YES" responses on the checklist and a paper's acceptance rate. However, this trend is not monotonic. Papers that claim "YES" for every single item often experience a drop in acceptance rates. This suggests a "bad faith" reporting phenomenon where authors may check boxes to satisfy requirements without actually providing the necessary information in the manuscript.

Acceptance rates across conferences as a function of the number of 'YES' responses on the reproducibility checklist.

The Data Collection Gap

A significant "data gap" exists in the current research landscape. Submissions introducing new datasets are 5% less likely to be accepted than those utilizing existing benchmarks. Furthermore, these papers receive reproducibility scores that are 2% lower on average. This indicates that the community lacks established norms for documenting new data collection and processing, or that reviewers penalize the inherent complexity of new data work.

Code Availability and Reviewer Perception

Providing a link to source code is the single most influential factor in a reviewer's perception of reproducibility, increasing scores by an average of 0.18 points. Despite this, only about 46% of submissions claim to have open-sourced their code at the time of submission. Many authors cite "crunch time" as a reason for omitting code or data, suggesting these tasks are often deprioritized as deadlines approach.

Reviewer Sentiment and Impact

The checklist is generally viewed positively by the community, with 77% of reviewers finding it useful for evaluating submissions. However, the impact of reproducibility scores on final acceptance is secondary to general recommendation scores.

Acceptance rate conditioned on reviewer-assigned reproducibility scores (AvgReprod) and general recommendation scores (AvgRec).

As shown in the figure above, while higher reproducibility scores (AvgReprod) correlate with higher acceptance, the general recommendation score (AvgRec) is a much steeper and more definitive predictor of a paper's success.

Conclusion and Future Directions

The Reproducibility Checklist has successfully increased the reporting of efficiency metrics and hyperparameters compared to pre-2020 standards. However, self-reporting bias remains a challenge. To further improve the field, the authors propose:

  1. Staggered Deadlines: Allowing a one-week grace period after the main paper deadline for submitting code and appendices.
  2. Specialized Data Checklists: Implementing specific prompts for data collection practices to bridge the "data gap."
  3. Public Accountability: Making checklist responses public for accepted papers to ensure transparency.

Summary updated

Abstract

Scientific progress in NLP rests on the reproducibility of researchers' claims. The *CL conferences created the NLP Reproducibility Checklist in 2020 to be completed by authors at submission to remind them of key information to include. We provide the first analysis of the Checklist by examining 10,405 anonymous responses to it. First, we find evidence of an increase in reporting of information on efficiency, validation performance, summary statistics, and hyperparameters after the Checklist's introduction. Further, we show acceptance rate grows for submissions with more Yes responses. We find that the 44% of submissions that gather new data are 5% less likely to be accepted than those that did not; the average reviewer-rated reproducibility of these submissions is also 2% lower relative to the rest. We find that only 46% of submissions claim to open-source their code, though submissions that do have 8% higher reproducibility score relative to those that do not, the most for any item. We discuss what can be inferred about the state of reproducibility in NLP, and provide a set of recommendations for future conferences, including: a) allowing submitting code and appendices one week after the deadline, and b) measuring dataset reproducibility by a checklist of data collection practices.
BibTeX
@inproceedings{magnusson2023reproducibility,
  author = {Ian Magnusson and Noah A. Smith and Jesse Dodge},
  title = {{Reproducibility in NLP: What Have We Learned from the Checklist?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=i4PiekE26jY},
}
Figure