Analysis of Summarization Evaluation Experiments

Marie-Josée GouletNAACL 2007

TL;DR

This paper meticulously analyzes 22 scientific papers on automatic text summarization evaluation, identifying and categorizing the myriad variables involved in these experiments. The core contribution is a comprehensive framework for understanding and standardizing how summarization evaluation results are presented, aiming to improve the rigor and comparability of research in this field, particularly for French texts.

What Problem Does This Paper Address?

The field of automatic text summarization has seen significant advancements, but a critical bottleneck remains: how we evaluate the quality of the summaries produced. Researchers often employ diverse methodologies, variables, and reporting styles, making it challenging to compare results across studies or to replicate experiments. This lack of standardization hinders progress and makes it difficult for the scientific community, especially the francophone community, to establish best practices. The paper addresses this by systematically dissecting existing summarization evaluation experiments to identify common variables, inconsistencies, and methodological problems, with the ultimate goal of providing guidelines for more consistent and informative reporting.

Methods

The authors conducted an in-depth analysis of 22 scientific papers on summarization evaluation published between 1961 and 2005. This involved meticulously noting experimental variables and classifying them into four main categories:

  1. Information about source texts: Characteristics of the original documents used for summarization.
  2. Information about automatic summaries being evaluated: Properties of the summaries generated by systems.
  3. Information about other summaries used in the evaluation process: Details on reference summaries (gold standards or baselines).
  4. Information about evaluation methods and criteria: How the summaries were assessed.

The analysis focused on identifying trends, inconsistencies, and potential methodological flaws within these variables.

Datasets

The paper does not introduce new datasets. Instead, it analyzes existing scientific papers that themselves used various datasets for their evaluations. The source texts analyzed in these papers were diverse, including scientific papers, technical reports, newspapers, book extracts, memos, and legal documents. The languages of the source texts were predominantly English, but also included French, Korean, and Japanese.

Key Insights and Ideas

  • The Wild West of Evaluation Variables: The sheer number and variety of variables used across studies are striking. This highlights the ad-hoc nature of summarization evaluation and the need for standardization.
  • Source Text Length: A Standardization Nightmare: The way source text length is reported is highly inconsistent, ranging from word counts to page numbers, and sometimes even providing ranges (shortest/longest) or averages. This makes direct comparison difficult.
  • Automatic Summary Length: Equally Inconsistent: Similar to source texts, the length of automatic summaries is reported in various ways, leading to further inconsistencies.
  • Extracts vs. Abstracts: A Crucial Distinction: The paper emphasizes that the type of summary (extract vs. abstract) is a critical variable that influences the choice of evaluation methods and criteria, as they are produced and serve different purposes.
  • The Elusive "Gold Standard": While gold standard summaries are often used for comparison, the number of human summarizers involved in their creation is frequently not reported. This is a significant omission, as the subjectivity of summarization means more annotators can increase reliability.
  • Ambiguity in Instructions: Instructions given to human summarizers for creating gold standards are often vague, using terms like "important" without clear explanation, which can lead to subjective interpretations.

Key Results

The paper's primary result is a detailed categorization and description of the variables employed in summarization evaluation experiments. It identifies:

  • Source Text Variables: Number of texts (often <100, with some exceptions), length (reported inconsistently), type (predominantly scientific papers, technical reports, newspapers), and language (mostly English).
  • Automatic Summary Variables: Total number of summaries evaluated (often <100, with some exceptions), number per source text, whether they are multi-document, length (inconsistently reported), type (predominantly extracts), and purpose (rarely specified).
  • Other Summaries Variables: The paper details seven types of information about comparison summaries, including number, type, length, number of human summarizers, instructions given, and their profiles.

The analysis reveals a lack of common agreement on how evaluation results should be presented, underscoring the need for guidelines.

Limitations and Future Work

The authors acknowledge that their analysis is based on papers published up to 2005. They plan to further analyze evaluation methods and criteria within their corpora and then develop recommendations for presenting summarization evaluation results. The paper also implicitly highlights the limitation that many studies do not provide sufficient detail about their evaluation setup, making it hard to fully understand or replicate their findings. Future work will focus on developing a French terminology for summarization evaluation results and providing guidelines for the design of future evaluation campaigns, potentially for French automatic summaries.

Summary updated

BibTeX
@inproceedings{goulet2007analysis,
  author = {Marie-Josée Goulet},
  title = {{Analysis of Summarization Evaluation Experiments}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure