Enhanced Word Representations for Bridging Anaphora Resolution

Yufang Hou

Abstract

Most current models of word representations (e.g., GloVe) have successfully captured fine-grained semantics. However, semantic similarity exhibited in these word embeddings is not suitable for resolving bridging anaphora, which requires the knowledge of associative similarity (i.e., relatedness) instead of semantic similarity information between synonyms or hypernyms. We create word embeddings (embeddings_PP) to capture such relatedness by exploring the syntactic structure of noun phrases. We demonstrate that using embeddings_PP alone achieves around 30% of accuracy for bridging anaphora resolution on the ISNotes corpus. Furthermore, we achieve a substantial gain over the state-of-the-art system for bridging antecedent selection.

Introduction

In Natural Language Processing, bridging anaphora resolution is the task of linking an "anaphor" to its "antecedent" when they refer to different but related entities. For example, in the sentence "Yet another political scandal is racking Japan... the chief cabinet secretary announced...", a human reader understands that "the secretary" belongs to the government of "Japan."

Standard word embeddings like GloVe or word2vec are built on the distributional hypothesis—words in similar contexts have similar meanings. Consequently, they excel at semantic similarity (e.g., "secretary" is similar to "minister"). However, bridging requires associative relatedness (e.g., "secretary" is related to "Japan"). Standard models often fail here because they might rank "minister" as the best antecedent for "secretary" simply because they are semantically similar, even though "Japan" is the correct logical anchor.

Methodology: The "PP" Suffix Trick

The core innovation of this work is a specialized embedding model called embeddings_PP. This model exploits the way bridging relations are naturally expressed in English through specific syntactic structures:

  1. Prepositional structures: "X of Y" (e.g., "the door of the car")
  2. Possessive structures: "Y's X" (e.g., "the company's chairman")

Bridging relations are inherently asymmetric: a "door" is part of a "car," but a "car" is not part of a "door." To capture this directionality, the author extracts noun pairs from a massive corpus and appends a _PP postfix to the head noun (the potential anaphor). For example, from the phrase "travelers in the train station," the model extracts the pair: travelers_PP – station. This allows the model to learn a distinct vector for a word when it acts as a "related entity" versus its standalone noun form.

Qualitative comparison of nearest neighbors in standard GloVe vs. embeddings_PP.

As shown in the table above, while standard GloVe finds synonyms (e.g., "president" → "vice"), the _PP suffix allows the model to find associative anchors (e.g., "president_PP" → "usa", "corporation").

Evaluation and Results

The model was trained using the GloVe toolkit on 197 million noun pairs from the Gigaword corpus and evaluated on the ISNotes corpus, which contains 663 bridging cases.

The results demonstrate that specialized embeddings provide a significant boost over general-purpose ones. Using embeddings_PP alone to select antecedents achieved 30.32% accuracy, a substantial jump over standard GloVe (19.00%). The importance of directionality was confirmed by removing the _PP suffix, which caused accuracy to drop to 22.17%.

Furthermore, when integrated into a complex Markov Logic Network (MLN) model (Hou et al., 2013), the embeddings_PP feature pushed the system's performance to a new state-of-the-art.

Performance of the MLN model when integrated with various word embeddings.

As seen in the results, the addition of embeddings_PP to the MLN model II yielded an accuracy of 45.85%, representing a 4.5% absolute improvement over the baseline MLN model.

Discussion and Conclusion

This research reinforces the distinction between similarity and relatedness. By restricting training data to specific syntactic patterns (X of Y), the model ignores general contextual noise and focuses on relational data. The _PP vectors effectively capture the "semantic roles" of nouns; for instance, president_PP becomes highly associated with countries or organizations, acting as a function that seeks its logical owner.

While the model performs well on structural relations, it still faces challenges with context-specific relations that require deeper situational awareness. Future work may involve looking beyond the "head" of a noun phrase, incorporating pre-nominal modifiers to provide even more granular clues for resolution.

Summary updated

Abstract

Most current models of word representations(e.g.,GloVe) have successfully captured fine-grained semantics. However, semantic similarity exhibited in these word embeddings is not suitable for resolving bridging anaphora, which requires the knowledge of associative similarity (i.e., relatedness) instead of semantic similarity information between synonyms or hypernyms. We create word embeddings (embeddings_PP) to capture such relatedness by exploring the syntactic structure of noun phrases. We demonstrate that using embeddings_PP alone achieves around 30% of accuracy for bridging anaphora resolution on the ISNotes corpus. Furthermore, we achieve a substantial gain over the state-of-the-art system (Hou et al., 2013) for bridging antecedent selection.
BibTeX
@inproceedings{hou2018enhanced,
  author = {Yufang Hou},
  title = {{Enhanced Word Representations for Bridging Anaphora Resolution}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=rkWlRm-ubr},
}
Figure