Abstract
One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.
Introduction
For years, the NLP community relied on static word embeddings like Word2Vec or GloVe to capture relationships through vector arithmetic (e.g., $king - man + woman \approx queen$). While effective, these models are inherently limited to capturing what can be inferred from word co-occurrence, often struggling with complex, non-linear relationships.
With the advent of Transformers, models like BERT demonstrated a massive leap in performance. However, the mechanism by which they store relational knowledge remained opaque. Unlike static embeddings, BERT is a "black box" where relations cannot be extracted via simple vector subtraction. This research investigates whether BERT acts as a superior "similarity engine" or if it possesses a deeper understanding of entity connections, and proposes a method to "tickle" this knowledge out of the model using linguistic prompts.
Methodology
The authors developed a three-step pipeline to induce relations without the need for manually defined rules:
- Candidate Discovery: Using a small set of seed pairs for a specific relation (e.g.,
(Tokyo, Japan)), the system searches a large corpus (Wikipedia) for sentences where both entities appear in close proximity. These sentences serve as potential "templates." - Template Filtering: To ensure quality, templates are ranked based on BERT's ability to predict a hidden tail word when provided with the head word within that specific sentence structure. This filters out noisy or irrelevant contexts.
- Fine-tuning for Classification: The highest-ranked templates are used to fine-tune BERT as a binary classifier. Given a full sentence (e.g., "The capital of [s] is [t]"), the model uses the
[CLS]token output to determine if the pair(s, t)represents a valid instance of the relation.
Experimental Setup
The methodology was evaluated across three diverse benchmarks:
- Google Analogy Test Set: 14 relation types covering semantic and syntactic categories.
- Bigger Analogy Test Set (BATS): 40 relations focusing on inflectional morphology, derivation, and lexicography.
- DiffVec: 36 relations centered on "commonsense" and attributive knowledge (e.g.,
Cause:Effect).
Results and Discussion
Overall Performance
BERT demonstrated significant improvements over traditional baselines in semantic and commonsense tasks. As shown in the performance summary, the $BERT^{max}_{100}$ variant consistently achieved high F1 scores on the Google and DiffVec datasets, outperforming both SVM-based and translation-based (Trans) baselines.
The "Encyclopedia" vs. "Dictionary" Split
A key finding of this study is the disparity in BERT's performance across different types of knowledge. BERT excels as an "encyclopedia," showing superior performance in factual relations such as capitals, currencies, and city-to-state mappings. However, it functions poorly as a "dictionary," frequently losing to simple vector-based models on morphological tasks like verb tense changes or pluralization.
On the Google dataset, BERT achieved an F1 score of 77.0, significantly higher than the SVM baseline of 57.5. Conversely, on BATS morphological tasks, BERT's performance was often less than half that of the baselines. For instance, in identifying "Verb 3rd person" forms, the baseline reached 97.0 F1, while BERT managed only 20.3.
Insights on Context and Naturalness
The research highlights that BERT's performance is highly sensitive to phrasing. The model performs better when using "natural" sentences extracted from a corpus rather than short, hand-crafted templates. This suggests that BERT relies heavily on the specific linguistic contexts it encountered during pre-training. Furthermore, the model can identify relations from implicit evidence; for example, it can infer a capital-of relationship from a sentence describing the location of an embassy.
Conclusion and Future Work
The primary limitation identified is the Morphological Gap. The authors hypothesize that because morphological relationships (e.g., "cat" to "cats") are rarely explicitly defined in natural text, BERT fails to learn a robust template for them. Future research may explore whether larger models like GPT-4 overcome this weakness or if the encyclopedic advantage simply scales further. Additionally, the reliance on large corpora for template discovery suggests potential challenges when applying this method to rare or niche relations.
Summary updated