Integrating Predictions from Neural-Network Relation Classifiers into Coreference and Bridging Resolution

Ina Roesiger, Maximilian Köper, Kim Anh Nguyen, Sabine Schulte im WaldeNAACL 2018

TL;DR

This paper explores how to improve coreference and bridging resolution by integrating semantic relation predictions from neural networks. While their approach shows promise for bridging resolution, it doesn't significantly boost performance for coreference resolution, suggesting that existing coreference systems already implicitly capture much of this semantic information.

What Problem Does This Paper Address?

Resolving anaphora (pronouns and noun phrases referring back to something previously mentioned) is a fundamental task in Natural Language Processing. Two specific types of anaphora are coreference, where a new expression refers to the exact same entity (e.g., "John went to the store. He bought milk."), and bridging resolution, where a new expression refers to an entity related to a previously mentioned one, but not identical (e.g., "I went into the room. The windows were broken." - rooms typically have windows).

Traditional approaches to these tasks often rely on knowledge bases like WordNet or Wikipedia to infer the semantic relationships needed to link anaphors to their antecedents. However, automatically inducing and integrating this semantic knowledge directly into resolution systems remains a challenge. This paper investigates whether state-of-the-art neural network classifiers, trained on semantic relation benchmarks, can provide valuable information to improve both coreference and bridging resolution systems.

Methods

The core idea is to use neural networks to predict the semantic relationship between pairs of noun phrases (anaphor-antecedent candidates). These predicted probabilities are then used as features in existing coreference and bridging resolution systems.

  1. Semantic Relation Classification: They trained a feed-forward neural network using word representations (from ConceptNet and later fastText for improved out-of-vocabulary handling). The network takes a pair of words as input and outputs probabilities for different semantic relations (e.g., meronymy, hypernymy, coordination, random/no relation). They experimented with different training setups, including handling lexical overlap between training and test sets.
  2. Integration into Resolvers:
    • Bridging Resolver: They re-implemented a rule-based bridging resolver and added new rules that leverage the predicted semantic relations. For instance, they introduced a rule to search for an antecedent within the last three sentences for certain NP structures, and also experimented with a cosine similarity threshold on the predicted relation probabilities.
  3. Coreference Resolver: They used the IMS HotCoref resolver, which allows for easy integration of new features. They added features based on the predicted semantic relation probabilities, including the highest predicted relation and binned cosine similarities.

Datasets

  • OntoNotes corpus: Used as the primary dataset for coreference resolution.
  • ISNotes corpus: A subset of OntoNotes used for bridging resolution, annotated with information status.
  • BLESS (Baroni and Lenci, 2011): Used for training and evaluating the semantic relation classifiers. It contains word pairs across six relations: co-hyponymy/coordination, attribute, meronymy, hypernymy, and random.
  • WordNet (Fellbaum, 1998): Used in the second experiment for inducing semantic relations.

Key Insights and Ideas

  • Bridging resolution is more sensitive to semantic relations: The paper's core hypothesis is that bridging resolution, which inherently relies on inferring relatedness rather than identity, would benefit more from explicit semantic relation information.
  • Meronymy is a strong predictor for bridging: The relation "meronymy" (part-whole relationship, e.g., "wheel" is a "car") consistently showed the most promise for improving bridging resolution performance.
  • Coreference resolvers might already capture implicit semantics: The lack of significant improvement in coreference resolution suggests that existing systems, especially those using word embeddings, might already implicitly encode the necessary semantic knowledge.
  • Lexical overlap can inflate relation classifier performance: When training and testing on word pairs that share words, the relation classifier's performance can be artificially high. Evaluating without lexical overlap provides a more realistic assessment.
  • The complexity of real-world relations: The distribution of relations in actual corpus data (e.g., meronymy appearing in coreference pairs) is more complex than initially hypothesized, requiring careful consideration.

Key Results

  • Bridging Resolution:
    • The best bridging setup, using the meronymy relation with a cosine threshold of 0.2, achieved an F1 score of 20.38 (compared to a baseline of 18.0). This represents a modest but notable improvement.
  • Adding semantic relation features to the bridging resolver generally improved performance, particularly for the meronymy relation.
  • Coreference Resolution:
    • Adding semantic relation features to the coreference resolver resulted in a decrease in performance (CoNLL score) in all tested configurations. This indicates that the proposed features did not help, and may have even hurt, the coreference system.
  • Intrinsic Evaluation of Relation Classifiers:
    • The relation classifiers achieved high accuracy (e.g., 87.8% on the BLESS test set without lexical overlap). However, performance varied significantly by relation, with hypernymy being particularly challenging.
  • When evaluated without lexical overlap, the accuracy dropped (e.g., to 58.6%), highlighting the impact of this factor.

Limitations and Future Work

  • Limited Improvement for Coreference: The primary limitation is the lack of improvement for coreference resolution. The authors suggest this might be due to the high baseline performance and implicit semantic knowledge in existing systems.
  • Data Annotation: The authors note that the relation labels in their datasets (OntoNotes, ISNotes) don't perfectly match their hypotheses, and annotating a portion of the data with explicit relations would be beneficial for a more direct assessment.
  • Directionality of Relations: For some relations (like hypernymy), the directionality is crucial (e.g., "city" -> "resident" is typical, but not vice-versa). The current system doesn't always handle this perfectly.
  • Future Work: The authors suggest further exploration into finding general cases of bridging and potentially adapting the relation classification to better align with the specific needs of coreference and bridging tasks. Investigating why coreference systems don't benefit from these explicit semantic features is also a key area.

Summary updated

BibTeX
@inproceedings{roesiger2018integrating,
  author = {Ina Roesiger and Maximilian Köper and Kim Anh Nguyen and Sabine Schulte im Walde},
  title = {{Integrating Predictions from Neural-Network Relation Classifiers into Coreference and Bridging Resolution}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure