Introduction
In natural language processing, coreference resolution—identifying that different expressions like "the CEO" and "she" refer to the same entity—is a well-established task. However, "bridging resolution" presents a subtler challenge. It involves linking an anaphor to an antecedent based on non-identical semantic or encyclopedic relations, such as "part-whole" or "attribute-of" connections. For instance, in the sentence, "In June, farmers held onto meat, milk and grain... The Communists froze prices instead," a human reader understands that "prices" refers to the prices of the aforementioned commodities. For machines, this implicit link is difficult to resolve because the terms are not synonyms.
Bridging resolution suffers from two primary hurdles: extreme data scarcity compared to standard coreference datasets, and a lack of model alignment in general-purpose architectures like BERT, which are not explicitly trained to recognize these associative relationships.
Methodology
To address these issues, we introduce PairSpanBERT, which builds upon the SpanBERT architecture. SpanBERT is specifically optimized for representing spans of text, making it a natural fit for entity-level tasks.
Pre-training with Associative Noun Objective (ANO)
The core innovation of PairSpanBERT is the Associative Noun Objective (ANO). To overcome data scarcity, we first automatically generate a large-scale "noisy" dataset from the Gigaword corpus using syntactic patterns (e.g., "X of Y") and knowledge graphs like ConceptNet (identifying relations like PartOf or HasA).
During pre-training, we specifically mask the "anchors" (antecedents) identified in this data. The model is then tasked with predicting the anaphor based on the contextual representations of the masked anchor. This forces the model to learn the specific linguistic contexts—such as certain verbs or prepositions—that signal a bridging relationship.
Multi-Task Learning Framework
PairSpanBERT is integrated into a Multi-Task Learning (MTL) framework. This architecture jointly performs coreference and bridging resolution, sharing a feed-forward neural network (FFNN) to process span representations generated by the encoder.
Datasets and Evaluation
The model was evaluated on three gold-standard bridging datasets:
- ISNotes: 50 documents from the Wall Street Journal.
- BASHI: 50 documents from the Wall Street Journal with distinct annotation guidelines.
- ARRAU RST: A larger, multi-genre dataset containing 413 documents.
Results and Discussion
PairSpanBERT (specifically the hybrid PSBERT(R) variant) achieved new state-of-the-art results across all benchmarks. In end-to-end settings, the model improved resolution F-scores by 2.3 points on ISNotes, 1.3 points on BASHI, and 1.5 points on ARRAU RST.
A key insight from our experiments is that pre-training on noisy noun pairs is significantly more effective than using those pairs as explicit features during fine-tuning. The self-supervised ANO task allows the model to "absorb" the underlying logic of bridging. Qualitatively, the model excels at resolving "bare NPs" (nouns without determiners), a category that typically poses significant challenges for rule-based systems.
Conclusion
PairSpanBERT demonstrates that specialized pre-training objectives can significantly enhance a model's ability to resolve complex semantic links. While currently optimized for bridging, the architecture's success suggests potential applications in other "link-heavy" tasks like relation extraction. Future work will explore incorporating broader knowledge bases, such as Wikidata, to further improve the model's common-sense reasoning regarding entity relationships.
Summary updated