Abstract
This research addresses the challenge of automatically assigning information status to discourse entities in conversational English. By leveraging morpho-syntactic and lexical features, a C4.5 decision tree was trained to classify noun phrases (NPs) as old, mediated, or new. The model achieved an overall accuracy of 79.5%, significantly outperforming a hand-crafted baseline (64.4%). Furthermore, binary classification experiments reached accuracies as high as 93.1%, particularly when distinguishing 'old' information from other categories.
Introduction
Published at EMNLP 2006, this study represents a foundational step in computational linguistics, specifically in teaching machines to understand the "givenness" of information in human conversation. The ability to distinguish between previously mentioned information and new concepts is vital for natural language understanding and generation.
Problem Definition
In human discourse, we constantly track the information status of entities to ensure efficient communication. Entities typically fall into three categories:
- Old: Entities already established in the discourse (e.g., "I saw a cat. The cat was black.").
- New: Entities introduced for the first time.
- Mediated: Entities that are new to the discourse but inferable via context or general knowledge (e.g., "I bought a car, but the engine is loud").
Accurate classification of these statuses is crucial for NLP tasks such as Anaphora Resolution and improving the prosody of Text-to-Speech systems, where "new" information typically receives greater acoustic emphasis.
Methodology
The study compared a baseline of hand-crafted linguistic rules against a machine learning approach using the J48 implementation of the C4.5 decision tree. The model was trained on a subset of the Switchboard corpus, consisting of 147 dialogues and over 64,000 manually annotated NPs.
The feature set focused on morpho-syntactic and lexical properties that signal information status:
Key features included:
- String Matching: Identifying if the exact NP or a substring appeared previously (represented by 'prev mention' features).
- Mention Time: Tracking whether an entity is on its first, second, or subsequent mention.
- NP Type & Length: Distinguishing between pronouns, proper nouns, and common nouns, as well as the number of words in the phrase.
- Determiner: Analyzing the use of definite ("the"), indefinite ("a"), or possessive ("my") articles.
- Grammatical Role: Identifying the NP's function, such as subject or object.
Results and Analysis
The decision tree model achieved a significant performance boost over the rule-based system, reaching 79.5% accuracy compared to the baseline's 64.4%.
Key Findings:
- Binary Success: The highest accuracy (93.1%) was achieved in the binary task of distinguishing "Old" from "Not-Old" (mediated and new combined). This suggests that "mediated" entities share more characteristics with "new" information than with "old" information in terms of their linguistic signaling.
- Learning Curve: As shown in the figure above, performance plateaus relatively early, around 5,000 training instances. This indicates that the current feature set, rather than the volume of data, is the primary bottleneck for further improvement.
- Class Disparities: The model excels at identifying "old" entities but struggles with "new" entities, which are frequently misclassified as "mediated" because both often appear for the first time in a text.
Discussion and Future Work
A primary limitation identified is the "World Knowledge Gap." The model struggles with mediated entities that require external context (e.g., recognizing "the Pope" as a unique, known entity). Future improvements could involve:
- Lexical Databases: Integrating WordNet or FrameNet to capture semantic relationships between entities.
- Dialogue Context: Incorporating speaker turns and more sophisticated distance measures to model how information remains salient in a listener's memory over time.
This work demonstrates that while morpho-syntactic features provide a strong foundation for tracking discourse entities, deep semantic understanding remains a frontier for future research.
Summary updated