Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research

Surangika Ranathunga, Nisansa De Silva, Dilith Jayakody, Aloka FernandoACL 2024

Sure, here's a blog-style summary of the paper "Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research":

TL;DR

This paper investigates the extent to which the NLP community embraces openness by analyzing artifact reuse and sharing in research papers. It reveals a significant disparity in artifact availability across languages and venues, with a concerning percentage of researchers not releasing their promised artifacts, hindering reproducibility and equitable progress.

What Problem Does This Paper Address?

The advancement of research, particularly in fields like NLP, heavily relies on the availability of shared artifacts such as code, software, and datasets. This openness is crucial for reproducing research, building upon existing work, and fostering a collaborative environment. However, the NLP community has long suspected a lack of consistent openness and a significant disparity in artifact availability, especially across different languages. Without quantifiable evidence, it's difficult to understand the true degree of openness and its benefits. This paper aims to fill that gap by providing empirical data on artifact reuse and sharing within the NLP research landscape.

Methods

The researchers employed a semi-automated approach to analyze a sample of NLP research papers from the ACL Anthology. They extracted paper titles and abstracts, then categorized papers by language, using a definition that accounts for over 6000 languages and is based on resources available on Hugging Face. Papers were further grouped by publication venue (Main ACL conferences/journals, LREC, and Other).

To assess artifact reuse, they analyzed papers to see if they used existing datasets, code, or language models. To measure artifact sharing, they identified papers that promised to release new artifacts and checked for the presence of repository URLs. A detailed analysis was then conducted to verify if these repositories actually contained the promised artifacts. They also analyzed the availability of datasets and language models on Hugging Face.

Datasets

The primary dataset for this study was a sample of NLP research papers from the ACL Anthology. The researchers also utilized data and code from previous studies (Ranathunga and de Silva, 2022; Blasi et al., 2022; Rohatgi, 2022) and collected new data for papers published after 2022. For language categorization, they adopted a list of 6419 languages previously established by Ranathunga and de Silva (2022), which was built upon the work of Joshi et al. (2020).

Key Insights and Ideas

  • Disappointing Openness: A significant portion of papers that promise to release artifacts do not actually make them publicly available, with some venues showing as high as 67% of papers failing to provide a link.
  • Language Disparity is Staggering: The availability of NLP artifacts (data, code, LMs) is heavily skewed towards high-resource languages, leaving low-resource languages significantly behind. This disparity is evident across all artifact types and publication venues.
  • Venue Matters: Different publication venues exhibit distinct patterns in artifact reuse and sharing. For instance, LREC, focused on language resources, sees more papers introducing new datasets, while Main ACL conferences show higher code reuse for new implementations.
  • Code is King (for Reuse): Code is the most frequently reused artifact, spanning from older toolkits to modern libraries like Hugging Face.
  • Hugging Face as a Hub: Hugging Face has emerged as a dominant platform for releasing and accessing language models, though its adoption varies across language classes.
  • Broken Links and Empty Repositories: A notable issue is the presence of broken repository links and repositories that do not contain the promised artifacts, further hindering reproducibility.

Key Results

  • Out of 355 analyzed papers, 98.9% reused some form of artifact from previous research.
  • In the Main venue, 44% of papers that produced new artifacts did not mention their release. This figure rose to 67% in the "Other" category.
  • LREC had the lowest percentage (33%) of papers not mentioning artifact release, but 36% of those promising data release did not provide a repository URL.
  • The analysis of recent trends (2020-2023) revealed a considerable portion of papers promising data release had "dead-ends" when trying to locate it, with this issue being more pronounced for low-resource languages.
  • GitHub is the most favored repository for releasing data and code, while Hugging Face is the preferred platform for LM releases.
  • The correlation between the class-wise LM and data availability on Hugging Face is very high (Pearson correlation of 0.9972).

Limitations and Future Work

The authors acknowledge several limitations:

  • Incomplete Coverage: The keyword-based filtering mechanism might have missed papers that didn't explicitly mention artifact availability in their abstract or title.
  • Manual Reading Errors: Despite efforts, manual reading of hundreds of papers could have introduced minor inaccuracies.
  • Repository Verification: The check for repository link liveness was performed only once, and the actual content of repositories was not exhaustively verified for completeness or functionality.
  • Limited Scope: The study focused on a fraction of papers published in the ACL Anthology and did not include all possible NLP venues or repositories.

The authors suggest that it would be interesting to repeat this experiment in 5-10 years to observe changes in openness trends. They also emphasize the importance of adhering to the "Bender Rule" for naming languages to improve discoverability. Their code is publicly released to facilitate future research in this area.

Summary updated

BibTeX
@inproceedings{ranathunga2024shoulders,
  author = {Surangika Ranathunga and Nisansa De Silva and Dilith Jayakody and Aloka Fernando},
  title = {{Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure