Automatically Learning Cognitive Status for Multi-Document Summarization of Newswire

Ani Nenkova, Advaith Siddharthan, Kathleen R. McKeownEMNLP 2005 · OpenReview

Abstract

Generating natural and concise multi-document summaries requires an understanding of the "cognitive status" of entities—specifically, whether a person mentioned is already known to the reader and their importance to the narrative. This paper presents a machine learning approach to automatically classify these distinctions by analyzing the syntactic and lexical forms used in news articles. By modeling references to people, the authors demonstrate that cognitive status can be learned with high accuracy, providing a foundation for more sophisticated summarization systems.

The Challenge of Referring Expressions

In multi-document summarization, extractive systems often struggle with referring expressions. A summary might redundantly describe a well-known public figure or, conversely, fail to provide necessary context for a minor character. This occurs because current systems lack a model of the reader’s mental state, or cognitive status.

In linguistics, cognitive status refers to a speaker's assumptions about the listener's prior knowledge. For instance, an educated reader likely knows who "Andrei Sakharov" is, making a lengthy biographical description unnecessary. Conversely, a "local shopkeeper" requires context. This research aims to bridge this gap by teaching machines to infer these statuses directly from source texts.

Methodology

The authors frame cognitive status as a classification problem focusing on two primary distinctions:

  1. Hearer-Old vs. Hearer-New: Is the person a public figure likely known to the audience, or a new character introduced in this story?
  2. Major vs. Minor: Is the person central to the narrative or a peripheral figure?

Feature Engineering and Learning

The pipeline begins with named entity recognition and coreference resolution to link various mentions of the same person (e.g., "Sakharov," "the physicist," "he"). The team extracted 18 specific features for each entity, including syntactic clues like the number of appositives and pronouns, and lexical clues derived from a large web corpus.

The 18 features used to train the classifiers, ranging from reference counts to bigram probabilities.

The researchers employed a Support Vector Machine (SMO) for the hearer-old/new distinction and a Decision Tree (J48) for major/minor classification. Training data was drawn from the Document Understanding Conference (DUC) datasets (2001–2004), supplemented by a large web corpus to build language models of typical media descriptions.

Key Results and Insights

The experiments revealed that journalists act as "expert feature engineers"; the way they introduce and refer to people provides high-fidelity signals of status. For example, early use of a pronoun or the omission of a description strongly suggests a "hearer-old" or "major" status.

The machine learning models significantly outperformed baseline predictions:

  • Major/Minor Classification: Achieved 96% accuracy using the J48 decision tree. This is particularly significant given that only about 5% of people mentioned in source articles typically appear in human-written summaries.
  • Hearer-Old/New Classification: Achieved 76% accuracy using SMO, representing a 22% improvement over the majority-class baseline.

Validation on "unseen" Newsblaster data confirmed the model's robustness, maintaining a stable 75% accuracy for the Hearer-Old/New classifier. Furthermore, human consensus on "hearer-old" status was high (87–91%), validating that general world knowledge is consistent enough to be modeled computationally for specific target audiences.

Conclusion and Future Directions

While the system demonstrates high accuracy, it remains sensitive to coreference resolution errors. Future work involves moving beyond binary classifications to a gradient model of importance and integrating these insights into a generation component. Such a component could dynamically rewrite summary sentences—stripping redundant descriptions for famous individuals while ensuring new characters are properly introduced—leading to more human-like, efficient summaries.

Summary updated

Abstract

Machine summaries can be improved by using knowledge about the cognitive status of news article referents. In this paper, we present an approach to automatically acquiring distinctions in cognitive status using machine learning over the forms of referring expressions appearing in the input. We focus on modeling references to people, both because news often revolve around people and because existing natural language tools for named entity identification are reliable. We examine two specific distinctions---whether a person in the news can be assumed to be known to a target audience (hearer-old vs hearer-new) and whether a person is a major character in the news story. We report on machine learning experiments that show that these distinctions can be learned with high accuracy, and validate our approach using human subjects.
BibTeX
@inproceedings{nenkova2005automatically,
  author = {Ani Nenkova and Advaith Siddharthan and Kathleen R. McKeown},
  title = {{Automatically Learning Cognitive Status for Multi-Document Summarization of Newswire}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  url = {https://openreview.net/forum?id=Bk-2hzWOWS},
}
Figure