The (Non)Utility of Predicate-Argument Frequencies for Pronoun Interpretation

Andrew Kehler, Douglas Appelt, Lara Taylor, Aleksandr SimmaNAACL 2004

TL;DR

This paper investigates whether statistics about how words are used together (predicate-argument frequencies) can improve pronoun interpretation systems. The authors found that while these statistics offer some predictive power, they provide little to no significant improvement over systems relying solely on standard morphosyntactic features, suggesting that current pronoun resolution models are already quite effective with the latter.

What Problem Does This Paper Address?

Interpreting pronouns (like "it," "he," "she") in text is a fundamental task in Natural Language Processing (NLP). Existing systems often rely on "morphosyntactic" features – grammatical properties like gender and number agreement, and the relative positions of pronouns and their potential antecedents. While these features are effective, researchers believe that true understanding requires deeper "world knowledge" and inference. The paper explores a potential shortcut: using statistics derived from how words co-occur in specific grammatical roles (e.g., which nouns are commonly the object of the verb "force"). The hypothesis is that these "predicate-argument statistics" could act as a proxy for world knowledge, helping to disambiguate pronoun references.

Methods

The researchers implemented and compared three main approaches to pronoun interpretation:

  • Maximum Entropy (MaxEnt) Model: This is a machine learning approach that learns weights for various features. They trained MaxEnt models in two configurations:
  • Using only standard morphosyntactic features.
  • Augmenting the morphosyntactic features with predicate-argument statistics.
  • Using predicate-argument statistics as a post-processing filter to re-rank potential antecedents.
  • Naive Bayes Model: Another machine learning classifier, used as a baseline.
  • Hobbs Algorithm: A rule-based, non-machine learning algorithm that searches for antecedents based on syntactic structure. This served as a baseline to understand the contribution of learned features.

The predicate-argument statistics were collected from a large corpus (TDT-2) and smoothed using Good-Turing smoothing. They experimented with different combinations of morphosyntactic features to see how predicate-argument statistics interacted with them.

Datasets

The primary dataset used was the Automatic Content Extraction (ACE) program corpus, specifically segments from news and newswires. This corpus was used for both training and testing. The training data contained 2773 annotated third-person pronouns, and the test data (February 2002 evaluation set) contained 762 annotated third-person pronouns. The pronouns annotated referred to specific entity types: PERSONS, ORGANIZATIONS, GEOPOLITICAL ENTITIES, LOCATIONS, and FACILITIES.

Key Insights and Ideas

  • Morphosyntax is Powerful: The results strongly suggest that standard morphosyntactic features are already very effective for pronoun resolution, capturing a significant portion of the necessary information.
  • Predicate-Argument Statistics Offer Limited Gains: Despite the theoretical appeal of using word co-occurrence statistics as a proxy for world knowledge, they provided minimal, and sometimes even negative, improvements when added to sophisticated MaxEnt models.
  • Integration vs. Post-processing: Both integrating predicate-argument statistics as features and using them as a post-processing filter yielded similar, limited benefits. The integrated approach sometimes even slightly hurt performance.
  • Data Sparsity Might Be a Factor, But Not the Whole Story: While the authors acknowledge that data sparsity (not seeing enough examples of certain word combinations) could be an issue, their error analysis, which included an attempt to gather more robust statistics using web search, still did not show significant gains.
  • "Textbook" Examples Are Rare: The paper suggests that the clear-cut examples often used to illustrate the need for world knowledge might not be representative of real-world discourse, where morphosyntactic cues are often sufficient.

Key Results

  • The best-performing MaxEnt system, using only morphosyntactic features, achieved an accuracy of 76.16% on the held-out training data and 75.72% on the blind test data.
  • Adding predicate-argument statistics as features to this system improved performance by less than 0.5%.
  • In some configurations, using predicate-argument statistics actually hurt performance compared to using morphosyntactic features alone.
  • The Hobbs algorithm, as a baseline without training, achieved 68.23% accuracy.
  • Naive Bayes models showed similar trends but generalized less well to the test data.

Limitations and Future Work

  • Limited Scope of Statistics: The study focused on specific types of predicate-argument relationships (subject-verb, verb-object, possessive-noun). Other types of semantic relationships might exist.
  • Shallow Parsing: The systems relied on shallow constituent parses, which are known to be errorful. More robust parsing could potentially improve results.
  • "Impossible" Cases: The evaluation included "impossible" cases where no correct antecedent was provided due to misparsing, affecting the overall accuracy metric.
  • Future Work: The authors suggest that future research could explore how to capture "world knowledge" more effectively, perhaps through different statistical approaches or by integrating external knowledge bases. They also note the need for methods that can improve performance without negatively impacting cases that are already correctly resolved.

Summary updated

BibTeX
@inproceedings{kehler2004non,
  author = {Andrew Kehler and Douglas Appelt and Lara Taylor and Aleksandr Simma},
  title = {{The (Non)Utility of Predicate-Argument Frequencies for Pronoun Interpretation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure