Improving Relation Extraction with Knowledge-attention

Pengfei Li, Kezhi Mao, Xuefeng Yang, Qi Li

Abstract

While attention mechanisms have been proven to be effective in many NLP tasks, majority of them are data-driven. We propose a novel knowledge-attention encoder which incorporates prior knowledge from external lexical resources into deep neural networks for relation extraction task. Furthermore, we present three effective ways of integrating knowledge-attention with self-attention to maximize the utilization of both knowledge and data. The proposed relation extraction system is end-to-end and fully attention-based. Experiment results show that the proposed knowledge-attention mechanism has complementary strengths with self-attention, and our integrated models outperform existing CNN, RNN, and self-attention based models. State-of-the-art performance is achieved on TACRED, a complex and large-scale relation extraction dataset.

Introduction

Relation Extraction (RE) is the task of identifying the semantic relationship between two entities in a sentence (e.g., recognizing that "James Dobson" is the founder of "Focus On The Family"). While modern deep learning models like Transformers are excellent at this, they are notoriously data-hungry and often struggle to generalize when training data is scarce or noisy. Most current attention mechanisms are purely "data-driven"—they learn patterns solely from the provided text. This paper addresses the gap between these "black-box" neural models and the vast wealth of human-curated linguistic knowledge (keywords, cue phrases, and semantic frames) that could help a model recognize complex relations more accurately.

Methods

The authors introduce a Knowledge-attention Encoder. Unlike standard self-attention, which models how words in a sentence relate to each other, this encoder focuses on how words relate to Relation Indicators—keywords and phrases extracted from linguistic databases like FrameNet and expanded via Thesaurus.com.

The Knowledge-attention mechanism calculates weights by comparing input text embeddings (Q) against relation indicators (K & V).

Integration Strategies

To maximize the utility of both prior knowledge and data-driven patterns, the paper proposes three ways to combine knowledge-attention with standard self-attention:

  • Multi-channel Attention (MCA): Processes data through two independent encoders and merges the features using a third attention layer.
  • Softmax Interpolation (SI): Runs two separate classifiers and combines their output probabilities using a weighted average ($\beta$). This allows for high controllability; for instance, increasing the weight of the knowledge channel can improve precision.
  • Knowledge-informed Self-attention (KISA): Injects the knowledge-attention directly into the internal heads of a self-attention layer.
Overview of the proposed relation extraction system, illustrating the integration of knowledge-attention with self-attention via MCA, SI, and KISA.

Results and Key Insights

The proposed models achieve state-of-the-art results on the TACRED dataset, a large-scale benchmark with over 106,000 sentences. The Softmax Interpolation (SI) model achieved an F1 score of 67.8, significantly outperforming the 66.5 F1 of the standard self-attention baseline.

Key findings include:

  • Complementary Strengths: Knowledge-attention acts like a high-precision rule-based system, while self-attention provides broad contextual recall. Merging them yields a "best of both worlds" performance.
  • Relational Subspaces: By using a multi-head structure for knowledge-attention, the model can jointly attend to different "subspaces" of a relation, which is crucial for complex sentences where multiple indicators might be present.
  • Generalization: On the smaller SemEval2010-Task8 dataset, knowledge-augmented models showed a statistically significant lead, suggesting that prior knowledge is particularly beneficial when data is limited.

Conclusion

By incorporating external lexical knowledge into the attention mechanism, the authors demonstrate that "human-in-the-loop" linguistic cues can significantly boost deep learning performance in relation extraction. While the model still faces challenges with very long-distance dependencies, future work aims to incorporate higher-level conceptual or encyclopedic knowledge from sources like Wikipedia to further enhance semantic understanding.

Summary updated

Abstract

While attention mechanisms have been proven to be effective in many NLP tasks, majority of them are data-driven. We propose a novel knowledge-attention encoder which incorporates prior knowledge from external lexical resources into deep neural networks for relation extraction task. Furthermore, we present three effective ways of integrating knowledge-attention with self-attention to maximize the utilization of both knowledge and data. The proposed relation extraction system is end-to-end and fully attention-based. Experiment results show that the proposed knowledge-attention mechanism has complementary strengths with self-attention, and our integrated models outperform existing CNN, RNN, and self-attention based models. State-of-the-art performance is achieved on TACRED, a complex and large-scale relation extraction dataset.
BibTeX
@inproceedings{li2019improving,
  author = {Pengfei Li and Kezhi Mao and Xuefeng Yang and Qi Li},
  title = {{Improving Relation Extraction with Knowledge-attention}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=BZb6M0lzvL},
}
Figure