Abstract
Transformer-based language models have shown remarkable success in various NLP tasks, yet their internal understanding of complex discourse phenomena remains under-explored. This paper investigates whether pre-trained models like BERT and RoBERTa inherently capture bridging inference—the ability to link non-identical entities through conceptual relationships. Through attention head analysis and a novel "Of-Cloze" probing task, we demonstrate that these models possess significant zero-shot knowledge of bridging relations, particularly within higher layers and specialized attention heads.
Introduction
Bridging inference is a critical component of human discourse, allowing readers to connect an anaphor (e.g., "the economy") to an antecedent (e.g., "Poland") via implicit relationships. Unlike coreference resolution, which deals with identity, bridging requires understanding semantic and commonsense links. This study probes whether such knowledge is emergent in large-scale language models without explicit supervision.
Methodology
The researchers employed two primary probing techniques:
- Attention Head Analysis: Examining the internal attention mechanisms of BERT-base to identify heads that naturally attend to bridging pairs. The "bridging signal" is calculated as the ratio of attention weight between anaphors and their corresponding antecedents.
- "Of-Cloze" Probing: A zero-shot task where the model predicts a masked antecedent in a prompt like
[Context] the economy of [MASK]. This evaluates the model's ability to leverage syntactic cues and semantic knowledge to resolve bridging links.
The experiments were conducted using the ISNotes corpus, which contains 663 manually annotated bridging anaphors across diverse relation types such as part-of and attribute-of.
Results and Discussion
The Hierarchy of Discourse Knowledge
Analysis reveals that bridging information is not uniformly distributed across the model. Lower layers primarily handle surface-level features, whereas higher layers exhibit increased activity related to discourse-level connections. This suggests that transformer models develop a hierarchical understanding of language, where complex referential relationships are resolved in the final stages of processing.
Specialized Attention Heads
Specific heads act as "discourse specialists." For instance, head 1 in layer 5 and head 12 in layer 9 consistently focus on bridging relations. These heads demonstrate a natural affinity for linking anaphors to their antecedents even without task-specific fine-tuning.
Zero-Shot Performance
Using the "Of-Cloze" test, RoBERTa-large achieved an accuracy of 28.05% on ISNotes. While lower than fine-tuned state-of-the-art models (~50%), this performance is significant for a zero-shot setting, outperforming traditional systems that rely on hand-crafted semantic features.
Linguistic Cues and Limitations
The preposition "of" serves as a vital trigger for bridging knowledge; removing it leads to a sharp decline in performance. Furthermore, models struggle with long-distance dependencies and complex commonsense reasoning (e.g., linking "OPEC" to "pool"), often defaulting to local semantic proximity rather than global discourse logic.
Conclusion
This research provides evidence that transformer models capture sophisticated discourse phenomena like bridging inference during pre-training. Identifying specialized "bridging heads" opens avenues for developing more discourse-aware NLP systems by leveraging these inherent signals.
Summary updated