Abstract
Generating natural and concise multi-document summaries requires an understanding of the "cognitive status" of entities—specifically, whether a person mentioned is already known to the reader and their importance to the narrative. This paper presents a machine learning approach to automatically classify these distinctions by analyzing the syntactic and lexical forms used in news articles. By modeling references to people, the authors demonstrate that cognitive status can be learned with high accuracy, providing a foundation for more sophisticated summarization systems.
The Challenge of Referring Expressions
In multi-document summarization, extractive systems often struggle with referring expressions. A summary might redundantly describe a well-known public figure or, conversely, fail to provide necessary context for a minor character. This occurs because current systems lack a model of the reader’s mental state, or cognitive status.
In linguistics, cognitive status refers to a speaker's assumptions about the listener's prior knowledge. For instance, an educated reader likely knows who "Andrei Sakharov" is, making a lengthy biographical description unnecessary. Conversely, a "local shopkeeper" requires context. This research aims to bridge this gap by teaching machines to infer these statuses directly from source texts.
Methodology
The authors frame cognitive status as a classification problem focusing on two primary distinctions:
- Hearer-Old vs. Hearer-New: Is the person a public figure likely known to the audience, or a new character introduced in this story?
- Major vs. Minor: Is the person central to the narrative or a peripheral figure?
Feature Engineering and Learning
The pipeline begins with named entity recognition and coreference resolution to link various mentions of the same person (e.g., "Sakharov," "the physicist," "he"). The team extracted 18 specific features for each entity, including syntactic clues like the number of appositives and pronouns, and lexical clues derived from a large web corpus.
The researchers employed a Support Vector Machine (SMO) for the hearer-old/new distinction and a Decision Tree (J48) for major/minor classification. Training data was drawn from the Document Understanding Conference (DUC) datasets (2001–2004), supplemented by a large web corpus to build language models of typical media descriptions.
Key Results and Insights
The experiments revealed that journalists act as "expert feature engineers"; the way they introduce and refer to people provides high-fidelity signals of status. For example, early use of a pronoun or the omission of a description strongly suggests a "hearer-old" or "major" status.
The machine learning models significantly outperformed baseline predictions:
- Major/Minor Classification: Achieved 96% accuracy using the J48 decision tree. This is particularly significant given that only about 5% of people mentioned in source articles typically appear in human-written summaries.
- Hearer-Old/New Classification: Achieved 76% accuracy using SMO, representing a 22% improvement over the majority-class baseline.
Validation on "unseen" Newsblaster data confirmed the model's robustness, maintaining a stable 75% accuracy for the Hearer-Old/New classifier. Furthermore, human consensus on "hearer-old" status was high (87–91%), validating that general world knowledge is consistent enough to be modeled computationally for specific target audiences.
Conclusion and Future Directions
While the system demonstrates high accuracy, it remains sensitive to coreference resolution errors. Future work involves moving beyond binary classifications to a gradient model of importance and integrating these insights into a generation component. Such a component could dynamically rewrite summary sentences—stripping redundant descriptions for famous individuals while ensuring new characters are properly introduced—leading to more human-like, efficient summaries.
Summary updated