Improving Relation Extraction with Knowledge-attention

Pengfei Li, Kezhi Mao, Xuefeng Yang, Qi Li

Abstract

While attention mechanisms have been proven to be effective in many NLP tasks, majority of them are data-driven. We propose a novel knowledge-attention encoder which incorporates prior knowledge from external lexical resources into deep neural networks for relation extraction task. Furthermore, we present three effective ways of integrating knowledge-attention with self-attention to maximize the utilization of both knowledge and data. The proposed relation extraction system is end-to-end and fully attention-based. Experiment results show that the proposed knowledge-attention mechanism has complementary strengths with self-attention, and our integrated models outperform existing CNN, RNN, and self-attention based models. State-of-the-art performance is achieved on TACRED, a complex and large-scale relation extraction dataset.

Introduction

Relation Extraction (RE) is a fundamental task in Information Extraction, aimed at identifying semantic relationships between entities within a sentence—for instance, determining that "James Dobson" is the founder of "Focus on the Family." While deep learning architectures like Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers have significantly advanced the field, they remain heavily dependent on large-scale, high-quality training data.

A primary challenge in RE is that data-driven models often struggle with complex or infrequent relations. Traditional feature-based methods relied on human-curated linguistic rules which, while precise, lacked scalability. Conversely, modern neural networks scale well but frequently overlook the specific linguistic cues (keywords and cue phrases) that human annotators use to identify relationships. This research introduces a Knowledge-attention Encoder to bridge this gap by injecting structured, external lexical knowledge directly into the attention mechanism of a Transformer-based model.

The Knowledge-attention Encoder

The proposed architecture operates by transforming text from a general semantic space into a specialized "relational semantic space." This is achieved through three primary components:

1. Relation Indicator Generation

The model leverages external linguistic resources to extract "relation indicators." These are keywords and cue phrases derived from FrameNet (a semantic frame database) and expanded using synonyms from Thesaurus.com. These indicators serve as the "top-down" linguistic priors for the model.

2. Knowledge-attention Mechanism

Unlike standard self-attention, which computes dependencies between words within the input sentence, the knowledge-attention mechanism calculates attention scores between input words and the pre-defined relation indicators.

The Knowledge-attention mechanism calculates scores between input words (Q) and relation indicators (K & V) to produce a relational hidden representation.

As shown in the figure above, the model subtracts the mean of the relation indicators from the hidden representations. This operation forces irrelevant words to have minimal outputs while pushing relevant words closer to their corresponding relation indicators in the vector space.

3. Integration Strategies

To maximize the benefits of both data-driven learning and linguistic knowledge, the authors propose three integration strategies:

  • Multi-channel Attention (MCA): Processes self-attention and knowledge-attention in separate streams, merging their feature vectors at the end.
  • Softmax Interpolation (SI): Generates independent predictions from both attention types and uses a weighted average ($\beta$) for the final result.
  • Knowledge-informed Self-attention (KISA): Directly integrates knowledge into every head of the self-attention mechanism, creating a unified representation.
The overall architecture of the Relation Encoder showing how input sentences and relation indicators are processed through integrated attention channels.

Experimental Evaluation

Datasets

The models were evaluated on two benchmarks:

  • TACRED: A large-scale dataset with over 106k sentences and 41 relation types. It is particularly challenging due to its high class imbalance (79.5% "no relation") and long sentence lengths.
  • SemEval2010-Task8: A smaller dataset with 9 directed relations, used to evaluate the model's generalization capabilities.

Key Results

The integrated models achieved state-of-the-art (SOTA) performance on the TACRED dataset with an F1 score of 67.9, surpassing previous Graph Convolutional Network (GCN) and CNN-based models. The knowledge-attention encoder alone outperformed standard CNN and RNN baselines by at least 1.3 F1 points.

Ablation studies highlighted the importance of the external knowledge base's breadth; specifically, the inclusion of synonyms from Thesaurus.com provided a significant performance boost. Furthermore, the Softmax Interpolation (SI) model allowed for fine-tuning the balance between precision and recall by adjusting the priority weight $\beta$.

Discussion and Future Work

The research demonstrates that knowledge-attention and self-attention have complementary strengths: knowledge-attention provides high precision by identifying specific linguistic markers, while self-attention provides high recall by capturing subtle data patterns.

Despite these gains, limitations remain regarding extremely long-distance dependencies and the confusion of semantically similar relations (e.g., "city of death" vs. "city of residence"). Future research will explore moving beyond lexical indicators toward conceptual knowledge from encyclopedic bases like Wikidata to capture higher-level semantics. The authors also intend to apply this knowledge-attention framework to broader NLP tasks such as sentiment analysis and question answering.

Summary updated

Abstract

While attention mechanisms have been proven to be effective in many NLP tasks, majority of them are data-driven. We propose a novel knowledge-attention encoder which incorporates prior knowledge from external lexical resources into deep neural networks for relation extraction task. Furthermore, we present three effective ways of integrating knowledge-attention with self-attention to maximize the utilization of both knowledge and data. The proposed relation extraction system is end-to-end and fully attention-based. Experiment results show that the proposed knowledge-attention mechanism has complementary strengths with self-attention, and our integrated models outperform existing CNN, RNN, and self-attention based models. State-of-the-art performance is achieved on TACRED, a complex and large-scale relation extraction dataset.
BibTeX
@inproceedings{li2019improving,
  author = {Pengfei Li and Kezhi Mao and Xuefeng Yang and Qi Li},
  title = {{Improving Relation Extraction with Knowledge-attention}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=BZb6M0lzvL},
}
Figure