Automated Text Summarization and the SUMMARIST System

Eduard H. Hovy, Chin-Yew LinTIPSTER 1998 · DOI

TL;DR

This paper introduces SUMMARIST, a modular system for automated text summarization, and proposes a typology of summaries to better understand the field. It details the system's architecture, which progresses from topic identification to interpretation and generation, and discusses novel methods for evaluating summary quality.

What Problem Does This Paper Address?

Automated text summarization aims to condense large amounts of text into shorter, informative versions. While interest in this field has surged with the growth of online information, a clear understanding of what constitutes a "good" summary and how to systematically build systems that produce them remains elusive. The authors identify a need for a structured approach to summarization, encompassing a clear definition of summary types, robust system design, and reliable evaluation metrics. They also highlight the challenge of moving beyond simple extractive summaries to more sophisticated abstractive ones, which requires incorporating world knowledge.

Methods

SUMMARIST is built with a modular design, allowing for experimentation and the integration of various techniques. The core process is broken down into three phases:

  • Topic Identification: This is the most developed phase, focusing on identifying the most important sentences in a text. It employs several modules, including:
  • Position Module: Leverages the observation that certain genres have predictable structures where important information is often found in specific positions (e.g., title, first sentence of paragraphs). It uses an "Optimal Position Policy" (OPP) derived from analyzing human-created abstracts.
  • Cue Phrase Module: Identifies phrases that signal important content (e.g., "in summary," "the best") or less important content.
  • Topic Signature Module: Uses word co-occurrence patterns to build "topic signatures" representing semantic concepts, which are then used to score sentences based on their relevance to these concepts.
  • Discourse Structure Module: Analyzes the hierarchical structure of text to identify and retain important clauses.
  • Topic Identification Integration Module: Combines scores from various modules using a combination function (initially hand-tuned, later learned via decision trees and perceptrons) to rank sentences.
  • Topic Interpretation: This phase aims to create abstracts by re-interpreting and fusing extracted topics into more succinct ones. This involves concept generalization (e.g., "apples, pears, oranges" -> "fruit") and script identification. The paper details experiments using WordNet for concept generalization and topic signatures for semantic concept representation.
  • Summary Generation: This phase produces the final output. For extracts, it's a null step. For abstracts, it involves reformulating the interpreted material into a coherent text. The authors plan to use existing language generators for this.

The paper also introduces a typology of summaries based on input characteristics (source size, domain specificity, genre/scale), output characteristics (derivation, coherence, partiality, conventionality), and usage (audience, usage type, expansiveness).

Datasets

  • Ziff-Davis texts: Approximately 13,000 newspaper articles announcing computer products, used for training the Optimal Position Policy (OPP).
  • Wall Street Journal (WSJ) corpus: A set of several thousand newspaper articles, used for training the OPP and for building topic signatures.
  • TIPSTER-SUMMAC summarization evaluation data: Used as training data for learning combination functions, specifically a portion of the dry run results annotated for relevance and popularity. A subsequent experiment used Q&A summary evaluation data from TIPSTER-SUMMAC, which contains essential text fragments judged by human annotators.

Key Insights and Ideas

  • A Typology is Crucial: The authors argue that a clear typology of summaries is essential for advancing the field, helping researchers and developers understand the variations and requirements of different summarization tasks.
  • Modular Design for Flexibility: SUMMARIST's modular architecture allows for independent development and testing of different summarization techniques, facilitating experimentation and adaptation.
  • Bridging Surface and Deep Processing: The system aims to combine robust NLP techniques with symbolic world knowledge to overcome the limitations of each approach alone, leading to higher quality summaries.
  • The Power of Position: The Position Module demonstrates that sentence position is a surprisingly strong indicator of importance, especially in certain genres, and can be effectively learned and applied.
  • World Knowledge is Key for Abstracts: Moving from extracts to abstracts necessitates incorporating world knowledge, which the authors explore through concept generalization and topic signatures.
  • Evaluation is a Major Hurdle: The paper highlights the difficulty of objectively evaluating summary quality, suggesting that current methods are still evolving and that task-specific evaluation is critical.

Key Results

  • Topic Identification: The Position Module, when tested on the Ziff-Davis corpus, achieved a 10-sentence extract (approx. 15% of text) that intersected with 95% of corresponding human abstracts.
  • Discourse Structure Module: Extracts generated using Marcu's discourse structure approach achieved an F-score of 68.86% for news articles and 70.42% for Scientific American texts, approximating human performance.
  • Combination Function Learning: Using a decision tree learning algorithm to combine module scores significantly improved SUMMARIST's performance on an unseen test set from 33.02% (Recall/Precision) to 58.07%, an improvement of 25%.
  • Evaluation Experiments: In a pilot test of the Shannon Game, students who saw full texts required an average of 11 additional letter guesses to recreate the text, while those who saw summaries required approximately 150, and those who saw no text required over 1,100.
  • Comparison of Evaluation Methods: Experiment 2 suggested that different evaluation methods (Shannon Game, Question Game, Classification Game) yield different results, and that summary types (e.g., extract vs. abstract) also affect these results. Notably, SUMMARIST extracts performed comparably to human abstracts in some evaluations.

Limitations and Future Work

  • World Knowledge Acquisition: The paper acknowledges that acquiring sufficient world knowledge for robust topic interpretation remains a significant challenge.
  • Microplanning and Generation: The authors have not yet focused on developing advanced language generation capabilities for SUMMARIST, which is crucial for producing high-quality abstracts.
  • Training Data Shortage: A major bottleneck for current summarization systems is the lack of good training data across various domains and genres, particularly triples of full text, human abstract, and corresponding extract.
  • Evaluation Nuances: The paper suggests that more extensive studies are needed to understand how different evaluation methods and summary types interact, and to establish reliable evaluation procedures.
  • Domain-Specific Techniques: While SUMMARIST is designed to be multilingual, the paper notes that domain-specific techniques could further improve performance.
  • Future Work: The authors plan to continue developing the SUMMARIST system, focusing on improving topic interpretation and generation, and conducting more extensive evaluations. They also suggest exploring smaller contexts for creating more accurate topic signatures.

Summary updated

BibTeX
@inproceedings{hovy1998automated,
  author = {Eduard H. Hovy and Chin-Yew Lin},
  title = {{Automated Text Summarization and the SUMMARIST System}},
  booktitle = {TIPSTER},
  year = {1998},
  doi = {10.3115/1119089.1119121},
}
Figure