Abstract
Most current models of word representations (e.g., GloVe) have successfully captured fine-grained semantics. However, semantic similarity exhibited in these word embeddings is not suitable for resolving bridging anaphora, which requires the knowledge of associative similarity (i.e., relatedness) instead of semantic similarity information between synonyms or hypernyms. We create word embeddings (embeddings_PP) to capture such relatedness by exploring the syntactic structure of noun phrases. We demonstrate that using embeddings_PP alone achieves around 30% of accuracy for bridging anaphora resolution on the ISNotes corpus. Furthermore, we achieve a substantial gain over the state-of-the-art system for bridging antecedent selection.
Introduction
In Natural Language Processing, bridging anaphora resolution is the task of linking an "anaphor" to its "antecedent" when they refer to different but related entities. For example, in the sentence "Yet another political scandal is racking Japan... the chief cabinet secretary announced...", a human reader understands that "the secretary" belongs to the government of "Japan."
Standard word embeddings like GloVe or word2vec are built on the distributional hypothesis—words in similar contexts have similar meanings. Consequently, they excel at semantic similarity (e.g., "secretary" is similar to "minister"). However, bridging requires associative relatedness (e.g., "secretary" is related to "Japan"). Standard models often fail here because they might rank "minister" as the best antecedent for "secretary" simply because they are semantically similar, even though "Japan" is the correct logical anchor.
Methodology: The "PP" Suffix Trick
The core innovation of this work is a specialized embedding model called embeddings_PP. This model exploits the way bridging relations are naturally expressed in English through specific syntactic structures:
- Prepositional structures: "X of Y" (e.g., "the door of the car")
- Possessive structures: "Y's X" (e.g., "the company's chairman")
Bridging relations are inherently asymmetric: a "door" is part of a "car," but a "car" is not part of a "door." To capture this directionality, the author extracts noun pairs from a massive corpus and appends a _PP postfix to the head noun (the potential anaphor). For example, from the phrase "travelers in the train station," the model extracts the pair: travelers_PP – station. This allows the model to learn a distinct vector for a word when it acts as a "related entity" versus its standalone noun form.
As shown in the table above, while standard GloVe finds synonyms (e.g., "president" → "vice"), the _PP suffix allows the model to find associative anchors (e.g., "president_PP" → "usa", "corporation").
Evaluation and Results
The model was trained using the GloVe toolkit on 197 million noun pairs from the Gigaword corpus and evaluated on the ISNotes corpus, which contains 663 bridging cases.
The results demonstrate that specialized embeddings provide a significant boost over general-purpose ones. Using embeddings_PP alone to select antecedents achieved 30.32% accuracy, a substantial jump over standard GloVe (19.00%). The importance of directionality was confirmed by removing the _PP suffix, which caused accuracy to drop to 22.17%.
Furthermore, when integrated into a complex Markov Logic Network (MLN) model (Hou et al., 2013), the embeddings_PP feature pushed the system's performance to a new state-of-the-art.
As seen in the results, the addition of embeddings_PP to the MLN model II yielded an accuracy of 45.85%, representing a 4.5% absolute improvement over the baseline MLN model.
Discussion and Conclusion
This research reinforces the distinction between similarity and relatedness. By restricting training data to specific syntactic patterns (X of Y), the model ignores general contextual noise and focuses on relational data. The _PP vectors effectively capture the "semantic roles" of nouns; for instance, president_PP becomes highly associated with countries or organizations, acting as a function that seeks its logical owner.
While the model performs well on structural relations, it still faces challenges with context-specific relations that require deeper situational awareness. Future work may involve looking beyond the "head" of a noun phrase, incorporating pre-nominal modifiers to provide even more granular clues for resolution.
Summary updated