Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization

Margot MieskesACL 2022

TL;DR

This paper investigates the replicability of research in automatic summarization, a field with seemingly ideal conditions for reproduction due to readily available datasets and evaluation metrics. Despite this, the authors find that replicating reported results is surprisingly difficult, highlighting that simply providing artifacts isn't enough; detailed reporting of all experimental parameters is crucial.

What Problem Does This Paper Address?

In scientific research, especially in fields like Natural Language Processing (NLP), the ability to replicate previous findings is paramount for building trust and advancing knowledge. However, a persistent challenge is the lack of complete reporting of experimental setups, including specific parameters, software versions, or implementation details. This makes it difficult for other researchers to reproduce results, even when datasets and evaluation methods are standard. The authors focus on automatic summarization, a sub-field where many conditions for replication appear to be met (e.g., standard datasets like DUC, well-known evaluation metrics like ROUGE), to explore why replication remains a challenge. They aim to understand what factors hinder or facilitate the replication of reported results in this seemingly "near-perfect" environment.

Methods

The authors employ a two-pronged approach:

  1. Literature Review: They analyze existing research on replicability in NLP, noting common challenges such as preprocessing steps, experimental setup, versioning, and system variations.
  2. Empirical Replication Experiments: They attempt to replicate reported results from specific papers in automatic summarization. This involves:
  3. Using standard datasets (DUC 2002 for Single-Document Summarization - SDS, and DUC 2004 for Multi-Document Summarization - MDS).
  4. Utilizing available summarization methods "out of the box" or retraining them.
  5. Paying close attention to the impact of various preprocessing parameters (e.g., stemming, stopword filtering) on evaluation metrics like ROUGE.
  6. Investigating specific systems and their reported results, comparing their own replicated results to those in the literature.

Datasets

The paper uses the following standard datasets for automatic summarization:

  • DUC 2002: Used for Single-Document Summarization (SDS) tasks. It contains over 500 documents from 59 thematic clusters, with a target summary length of 100 words.
  • DUC 2004: Used for Multi-Document Summarization (MDS) tasks. It contains 500 documents from 50 thematic clusters, with a length restriction of 665 bytes (approximately 100 words for English).
  • Hong et al. (2014) dataset: A dataset published by Hong et al. (2014) containing summaries from various methods, which the authors use to replicate their results.

Key Insights and Ideas

  • "Near-perfect" is not perfect: Even in a domain with standardized datasets and evaluation metrics like automatic summarization, replicating results is far from straightforward.
  • Parameter reporting is crucial: The study strongly suggests that simply providing code or data is insufficient. Detailed reporting of all experimental parameters, including those for preprocessing and evaluation (like ROUGE parameters), is essential for successful replication.
  • Preprocessing has a significant impact: The choice of preprocessing steps, such as stemming and stopword filtering, can lead to substantial differences in evaluation scores and rankings, sometimes even contradicting findings in the original papers.
  • Discrepancies in baseline reporting: The authors observed inconsistencies in how baseline results were reported across different papers, further complicating comparisons.
  • Replication is more than just running code: Even when system implementations are available, differences in experimental setup, parameter choices, and evaluation procedures can lead to significant variations in results, making direct replication challenging.

Key Results

  • SDS Replication: The authors found considerable discrepancies between officially reported results and those reported in the literature for DUC 2002. While some rankings were similar, ROUGE scores varied significantly. They observed that stemming generally led to better results, while stopword filtering alone led to worse results.
  • MDS Replication: For DUC 2004, the situation was slightly better, with less discrepancy in the top-ranked systems. However, when replicating results using various ROUGE parameter settings, they still observed considerable variation. Stemming again tended to yield results closer to official reports.
  • Impact of Parameters: Experiments with the Hong et al. (2014) dataset showed that while many cited papers report evaluation parameters, subtle differences (e.g., in length constraints or specific ROUGE settings) can lead to discrepancies.
  • MEAD and SVM: Experiments with the MEAD system and a retrained SVM model also showed that results can vary considerably depending on the parameters used, even when the system implementation is available.
  • Publication Replicability: The authors note that only about 40% of the publications they analyzed reported the full set of evaluation parameters, and almost 50% did not mention them at all, making replication unnecessarily complicated.

Limitations and Future Work

  • Hardware/Software Differences: The authors acknowledge that some observed differences in replication might be due to variations in hardware or software environments, although they attempted to control for this.
  • Limited Scope: The study focuses specifically on automatic summarization. Broader analyses across other NLP sub-fields would strengthen the findings.
  • Parameter Space Complexity: The vast parameter space in modern machine learning approaches, which are often unreported, presents a significant challenge for replication.
  • Future Work: The authors suggest that a next step is to re-evaluate their experiments using tools like py-rouge, which might offer different parameter options. They also emphasize the need for more comprehensive reporting of all parameters used in creating and evaluating NLP systems to ensure replicability and reproducibility.

Summary updated

BibTeX
@inproceedings{mieskes2022replicability,
  author = {Margot Mieskes},
  title = {{Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure