TL;DR
This paper presents a longitudinal study tracking NLP and ML researchers' views and experiences with reproducibility over two years (2022 and 2024). The findings indicate a positive trend, with researchers increasingly valuing and engaging in reproducibility efforts, though challenges with time, funding, and practical implementation persist.
What Problem Does This Paper Address?
Reproducibility is a cornerstone of scientific progress, ensuring that research findings can be independently verified. In fields like Natural Language Processing (NLP) and Machine Learning (ML), however, poor reproducibility is a persistent concern. This can stem from variations in experimental setups, evaluation data, or simply inadequate reporting of methods. While various initiatives like conference checklists and reproducibility statements have emerged, their actual impact on improving practices and understanding remains unclear. This paper aims to bridge this knowledge gap by systematically tracking how NLP and ML researchers' perceptions and experiences of reproducibility have evolved over time.
Methods
The study employs a longitudinal survey design, conducting two identical surveys of NLP and ML researchers in 2022 and 2024. The surveys were carefully designed using established principles to ensure clarity and shared meaning of questions. The 2024 survey included 31 questions covering participant demographics, experiences with recreating research (both their own and others'), recreating human evaluations, and barriers to reproducibility. The analysis compares the results from the two survey waves to identify changes in views and practices. Statistical methods like the Mann-Whitney U test and Chi-squared tests were used to assess significance. A high-level recoded analysis was also performed to assess changes in three constructs: reproduction activity, enabling reproducibility practices, and awareness of reproducibility.
Datasets
Not discussed in the paper. The study relies on survey responses from NLP and ML researchers, not on specific datasets for training or evaluation of models.
Key Insights and Ideas
- Growing Awareness and Engagement: There's a clear upward trend in researchers valuing reproducibility and actively engaging in reproduction efforts. This suggests that the ongoing discourse and initiatives around reproducibility are having a positive impact.
- Persistent Practical Barriers: Despite increased awareness, significant hurdles remain. Lack of time and funding are consistently cited as the primary obstacles to conducting reproducibility work.
- "Recreating Own Work" is Easier: Researchers find it generally easier to recreate their own past work compared to that of others, likely due to better access to their own code, data, and understanding of the experimental setup.
- Human Evaluation Reproducibility Challenges: Recreating human evaluations presents unique difficulties, with obtaining design details and experimental conditions being particularly problematic.
- The "Siren Song" of Checklists: While conferences are implementing reproducibility checklists, the study hints that simply checking boxes might not translate to genuine improvements in reproducibility. The reliance on self-reporting is a concern.
- Junior Researchers Lead in Sharing: More junior researchers (PhD students and those with less experience) are more likely to share their code, suggesting a potential cultural shift or greater emphasis on open practices in newer cohorts.
Key Results
- Positive Trend in Reproducibility Activity (HQ1): A statistically significant increase was observed in researchers engaging in reproduction efforts between 2022 and 2024.
- Increased Value of Reproducibility (HQ3): Researchers are more aware of and value reproducibility more in 2024 compared to 2022.
- Enabling Practices Lagging: While there was an increase in researchers taking steps to make their work reproducible (HQ2), this change was not statistically significant, suggesting that practical implementation remains a challenge.
- Recreating Others' Work: Nearly 5/6 of respondents had attempted to recreate others' research in the past 5 years, but only 42% consistently obtained similar results.
- Barriers to Reproduction: 64% of participants agreed they would submit more reproduction studies if conferences reserved acceptance rates for them, and 60% would do so if funding was available.
- Sharing Practices: 77% of participants always/frequently make model code available, and 76% make data available.
Limitations and Future Work
The authors acknowledge potential sampling bias, meaning the results might not fully represent the entire NLP/ML community, as researchers interested in reproducibility might be more inclined to participate. Geographical diversity was also limited due to the survey being conducted in English. Future work could explore these limitations further. The study suggests that addressing the practical barriers of time and funding, alongside developing better toolkits and methodologies for reproducible research, are crucial next steps. The findings also highlight the need for funders and conference organizers to actively support and incentivize reproducibility efforts.
Summary updated