TL;DR
This paper proposes a simple yet effective method to improve relation extraction (RE) by incorporating pre-trained knowledge base (KB) graph embeddings into a BERT-based model. The authors demonstrate that this approach significantly boosts performance on standard RE datasets, achieving competitive results with state-of-the-art methods that require more complex on-task training of graph embeddings.
What Problem Does This Paper Address?
Relation Extraction (RE) is a crucial task in Natural Language Processing (NLP) that aims to identify semantic relationships between entities in text. While large amounts of unstructured text data are available, structured data from Knowledge Bases (KBs) like Wikidata also offer valuable information. Previous work has explored using KBs to improve RE, often by dynamically learning KB entity representations alongside text. However, these methods can be complex and computationally expensive. This paper addresses the question of whether simply adding pre-computed, static KB graph embeddings to existing powerful language models like BERT can provide a significant performance boost for RE without the added complexity.
Methods
The core of the proposed method involves augmenting a standard BERT-based RE model with pre-trained KB graph embeddings. Specifically:
- Pre-trained Language Model: The authors use a BERT (or RoBERTa) encoder, which is fine-tuned on the RE task. The encoder is not frozen, allowing gradients to propagate and optimize token representations.
- Entity Filter: For multi-token entities, the average of the token embeddings is used as the entity's representation.
- Graph Embedding Augmentation: The key innovation is concatenating the pre-trained graph embedding of an entity (obtained from Lerer et al., 2019) to its BERT-based representation. This combined representation is then fed into the RE module.
- RE Module: This module, based on existing work (Dozat and Manning, 2017; Nguyen and Verspoor, 2019; Giorgi et al., 2019), first performs head-tail decompositions of the input representations and then uses a biaffine attention layer for relation classification.
The authors emphasize the simplicity of this approach, contrasting it with more complex methods that require on-task training of graph embeddings.
Datasets
The paper evaluates the proposed model on two popular RE datasets:
- Wikidata: A distant supervision dataset aligned with the Wikidata Knowledge Base.
- NYT: The New York Times corpus, also built using distant supervision and aligned with Freebase (and subsequently mapped to Wikidata).
For pre-trained graph embeddings, the authors utilize those provided by Lerer et al. (2019), which were trained on a "truthy" Wikidata dump using the TransE approach with a 200-dimensional embedding space.
Key Insights and Ideas
- Simplicity is Powerful: The most striking insight is that a straightforward approach of adding pre-trained KB graph embeddings to a BERT model yields significant performance improvements, rivaling more complex methods.
- Transferability of Topological Information: The study suggests that pre-trained graph embeddings effectively transfer topological information from the KB to the RE task, enhancing the model's understanding of entity relationships.
- Power-Law Scaling of Variance: The authors observe an interesting power-law relationship between the variance of F1 scores and the support (number of training examples) for a given relation. This suggests that a certain threshold of support (around 1000 examples) is crucial for achieving good performance with low uncertainty.
- Impact on Relation Classes: While not all relation classes benefit equally, the majority show consistent improvement with the addition of graph embeddings, leading to an overall boost in performance.
Key Results
- Wikidata: The model with graph embeddings achieved a performance boost of approximately 2-4% in micro-averaged F1 score compared to the baseline BERT model. While not always surpassing the absolute state-of-the-art in micro-averaged metrics, it achieved competitive results.
- NYT: The model achieved a significant improvement of about 3% in both micro- and macro-F1 scores. Crucially, it set a new state-of-the-art performance on this dataset for P@10 and P@30 metrics.
- Variance Reduction: The inclusion of graph embeddings generally led to a reduction in the variance of F1 scores across different runs, indicating more stable and reliable performance.
Limitations and Future Work
- Entity Identification Dependency: A significant limitation acknowledged by the authors is the reliance on correctly identified entities (gold entities). The task of Named Entity Recognition (NER) and Entity Linking (mapping entities to KB nodes) is itself challenging and currently limits the practical applicability of such models.
- Generalizability of Power-Law Finding: The observed power-law scaling of variance with support was found on the Wikidata dataset. Further investigation is needed to explore the generality of this finding across different datasets and relation types.
- Future Work: The authors suggest that future research should focus on addressing the entity identification challenge. They also propose that the adoption of pre-computed graph embeddings could become as widespread as pre-trained word embeddings, potentially benefiting a wider range of NLP tasks.
Summary updated