Unsupervised Learning of Contextual Role Knowledge for Coreference Resolution

David Bean, Ellen RiloffNAACL 2004

TL;DR

This paper introduces BABAR, a novel coreference resolution system that leverages "contextual role knowledge" learned unsupervisedly from text. By understanding the roles noun phrases play in events, BABAR significantly improves coreference resolution, especially for pronouns, without requiring manually annotated training data.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying mentions in text that refer to the same real-world entity (e.g., "John," "he," and "the president" all referring to the same person), is a fundamental challenge in Natural Language Processing. Traditional approaches often rely on lexical, syntactic, and semantic features of the mentions themselves. However, the authors argue that understanding the contextual roles these mentions play within events or relationships is crucial for accurate resolution. For instance, distinguishing between "kidnapped victims" and "kidnappers" in sentences like "After they were released..." versus "After they blindfolded the men..." hinges on knowing the typical roles associated with these groups. Existing methods often lack a robust way to capture and utilize this contextual role knowledge, especially in an unsupervised manner.

Methods

BABAR's core innovation lies in its unsupervised learning of contextual role knowledge. The system employs the following key techniques:

  • Reliable Case Resolution: To generate training data without manual annotation, BABAR first identifies anaphora that can be easily resolved using:
  • Lexical Seeding: Matching proper names (e.g., "George Bush" and "Bush") and existential noun phrases (e.g., "the FBI" appearing multiple times).
  • Syntactic Seeding: Utilizing a set of seven syntactic heuristics (e.g., reflexive pronouns, relative pronouns, appositives) to find straightforward anaphor-antecedent pairs.
  • Contextual Role Representation: The system uses information extraction patterns to represent contextual roles. These patterns are normalized into "caseframes," which approximate simple case frames capturing thematic roles (e.g., "<agent> kidnapped") or predicate-argument relations (e.g., "kidnapped for <NP>").
  • Four Contextual Role Knowledge Sources: BABAR learns four types of knowledge from the reliable case resolutions:
  • Caseframe Network (CFNet): Identifies caseframes that frequently co-occur in resolutions, suggesting a conceptual relationship between events.
  • Lexical Caseframe Expectations (CFLex): Learns which nouns typically co-occur with specific caseframes, enabling checks for substitutability.
  • Semantic Caseframe Expectations (WordSem-CFSem): Learns semantic classes that co-occur with caseframes, used to rule out candidates with mutually exclusive semantic classes.
  • CFSem-CFSem: Compares semantic expectations of caseframes for an anaphor and its candidate antecedent.
  • Dempster-Shafer Probabilistic Model: BABAR combines evidence from the four contextual role knowledge sources with seven general knowledge sources (gender, number, scope, lexical, recency, syntactic role) using a Dempster-Shafer model. This model is chosen for its principled way of combining evidence from multiple sources and its ability to handle uncertainty and competing hypotheses by redistributing belief across sets of possibilities.

Datasets

  • Training Data: The system was trained on the MUC-4 terrorism corpus and Reuters texts about natural disasters. The paper notes that larger corpora (MUC-4 and Reuters) were used for training compared to the smaller datasets available in MUC-6 and MUC-7, which were deemed insufficient for learning reliable co-occurrence statistics.
  • Test Data: For evaluation, the authors created blind test sets by manually annotating 40 documents for each domain (terrorism and natural disasters). These test sets contained 322 anaphoric links for terrorism and 447 for natural disasters.

Key Insights and Ideas

  • Contextual Roles are Crucial: The paper strongly suggests that understanding the role a noun phrase plays in an event is as important, if not more so, than its intrinsic features for coreference resolution.
  • Unsupervised Learning is Viable: BABAR demonstrates that valuable contextual role knowledge can be learned automatically from plain text without the need for expensive, manually annotated coreference data.
  • Pronouns Benefit Most: Contextual role knowledge proved particularly beneficial for resolving pronouns, which are semantically weak and heavily rely on context. This highlights a potential need for differentiated coreference resolution strategies based on anaphor type.
  • Synergistic Knowledge Sources: Combining multiple contextual role knowledge sources, along with general knowledge, leads to the greatest performance gains, suggesting that these sources capture complementary information and act synergistically within the Dempster-Shafer framework.
  • Dempster-Shafer for Robustness: The Dempster-Shafer model provides a principled way to integrate diverse evidence and gracefully handles situations where no single antecedent is clearly superior, opting for non-resolution rather than a potentially incorrect decision.

Key Results

  • BABAR achieved good performance in both the terrorism and natural disaster domains.
  • The addition of contextual role knowledge sources improved coreference performance, particularly for pronouns.
  • For pronouns, recall increased by +13% (terrorism) and +15% (disasters) with only a small decrease in precision.
  • For definite noun phrases, the impact was less pronounced, with some recall gains but also some precision drops.
  • When all four contextual role knowledge sources were used together, the greatest performance gains were observed.

Limitations and Future Work

  • Domain Specificity: The learned contextual role knowledge is domain-specific. While the paper uses two domains, extending this to a broader range of domains or developing domain-independent contextual role learning would be valuable.
  • Approximation of Caseframes: The paper acknowledges that the learned caseframes are a "crude approximation" of true thematic roles, as they are derived syntactically rather than semantically.
  • Future Work: The authors plan to:
  • Follow up on their approach and investigate other ways contextual role knowledge can be used.
  • Investigate other ways that contextual role knowledge can be used.
  • Potentially extend the resolution of existential NPs to cross-document scenarios.

Summary updated

BibTeX
@inproceedings{bean2004unsupervised,
  author = {David Bean and Ellen Riloff},
  title = {{Unsupervised Learning of Contextual Role Knowledge for Coreference Resolution}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure