The Fourth Workshop on Insights from Negative Results in NLP

Shabnam Tafreshi, Arjun Akula, João Sedoc, Aleksandr Drozd, Anna Rogers, Anna RumshiskyOpenalex Unknown 2023 · DOI · OpenAlex

Abstract

We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more or less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction. Furthermore, all but one of the experiments we selected for reproduction were discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a "reproduce" approach to a "standardize-then-reproduce-twice" approach. Our overall finding that the great majority of human evaluations in NLP is not repeatable, not reproducible, or too flawed to justify reproduction paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.

Introduction

In Natural Language Processing (NLP), human evaluation is considered the ultimate litmus test. While automatic metrics like BLEU or ROUGE are widely used, their validity is predicated on their correlation with human judgment. However, if the human evaluations themselves cannot be reproduced, the foundation of the field's evaluation metrics is on shaky ground. This study set out to conduct a "multi-lab multi-test" (MLMT) study to identify factors—such as evaluator expertise or sample size—that affect reproducibility. Instead of a systematic measurement of noise, the researchers encountered a systemic failure in research transparency and data retention.

Methodology

The research team followed a rigorous, five-stage filtering process to identify candidates for reproduction:

  1. Search: Scanned the ACL Anthology (2018–2022) for papers mentioning "human evaluation" and "participants," identifying 177 candidates.
  2. Suitability Check: Filtered out surveys, longitudinal studies, and experiments with prohibitive costs (>$2,000), leaving 116 papers.
  3. Author Correspondence: Contacted authors of the 116 papers to request original system outputs and evaluation interfaces.
  4. Detailed Annotation: For papers where data was provided, the team analyzed 16 experimental properties, including instructions, native speaker status, and task complexity.
  5. Selection: Attempted to select a balanced set of experiments for reproduction across multiple labs.

Key Insights: The Barriers to Reproducibility

The investigation revealed several recurring obstacles that prevent the verification of NLP research:

  • The "Vanishing Graduate Student" Problem: The most frequent reason for data unavailability was personnel turnover. Lead researchers (often graduate students) had moved on, and the underlying data files were lost or deleted.
  • Commercial Secrecy: Authors from commercial research groups occasionally refused to share evaluation details for proprietary reasons, rendering their peer-reviewed results unverifiable by the scientific community.
  • The "Expert" Illusion: Many papers claimed to use "expert" annotators. Upon closer inspection, these often turned out to be the authors themselves or immediate colleagues, introducing significant potential for bias.
  • Broken Randomization: In several instances, claims of randomized item ordering were contradicted by the raw data, which showed randomization was either applied incorrectly or omitted entirely.

Results and Findings

The findings paint a concerning picture of the current state of human evaluation in NLP:

  • Low Engagement: Only 39% of authors responded to requests for information regarding their published work.
  • The 13% Reality: Out of the 116 suitable papers, only 20 authors (representing 28 experiments) provided enough detail to progress to the annotation stage. Ultimately, only 13% of the initial pool met the basic criteria for reproduction.
  • Prevalent Experimental Flaws: Of the final six experiments selected for actual reproduction, five were found to have major flaws—such as mismatched item counts or errors in reported figures—that made reproduction attempts meaningless.
  • The Methodological Pivot: Because the team could not find enough valid experiments to reproduce, the study design was forced to change. The researchers moved to a "standardize-then-reproduce-twice" model, where they first fix the original experiment's flaws before attempting to measure its reproducibility.

Conclusion and Future Directions

The primary limitation of this study was the small sample size, dictated by the lack of author cooperation and the high rate of experimental errors. However, these negative results are highly informative. The authors strongly advocate for the adoption of Human Evaluation Datasheets (HEDS). Similar to model cards or data statements, HEDS would require researchers to document every detail of their human trials—from exact instructions to participant compensation—at the time of publication. Without a shift toward such standardization and rigorous data retention, human evaluation in NLP remains a "black box" that lacks the transparency required for scientific progress.

Summary updated

Abstract

We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible.We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable.As a result, we had to change our coordinated study design from a reproduce approach to a standardisethen-reproduce-twice approach.Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.
BibTeX
@inproceedings{tafreshi2023fourth,
  author = {Shabnam Tafreshi and Arjun Akula and João Sedoc and Aleksandr Drozd and Anna Rogers and Anna Rumshisky},
  title = {{The Fourth Workshop on Insights from Negative Results in NLP}},
  booktitle = {Openalex Unknown},
  year = {2023},
  doi = {10.18653/v1/2023.insights-1},
}
Figure