Abstract
While attention mechanisms have been proven to be effective in many NLP tasks, majority of them are data-driven. We propose a novel knowledge-attention encoder which incorporates prior knowledge from external lexical resources into deep neural networks for relation extraction task. Furthermore, we present three effective ways of integrating knowledge-attention with self-attention to maximize the utilization of both knowledge and data. The proposed relation extraction system is end-to-end and fully attention-based. Experiment results show that the proposed knowledge-attention mechanism has complementary strengths with self-attention, and our integrated models outperform existing CNN, RNN, and self-attention based models. State-of-the-art performance is achieved on TACRED, a complex and large-scale relation extraction dataset.
Introduction
Relation Extraction (RE) is the task of identifying the semantic relationship between two entities in a sentence (e.g., recognizing that "James Dobson" is the founder of "Focus On The Family"). While modern deep learning models like Transformers are excellent at this, they are notoriously data-hungry and often struggle to generalize when training data is scarce or noisy. Most current attention mechanisms are purely "data-driven"—they learn patterns solely from the provided text. This paper addresses the gap between these "black-box" neural models and the vast wealth of human-curated linguistic knowledge (keywords, cue phrases, and semantic frames) that could help a model recognize complex relations more accurately.
Methods
The authors introduce a Knowledge-attention Encoder. Unlike standard self-attention, which models how words in a sentence relate to each other, this encoder focuses on how words relate to Relation Indicators—keywords and phrases extracted from linguistic databases like FrameNet and expanded via Thesaurus.com.
Integration Strategies
To maximize the utility of both prior knowledge and data-driven patterns, the paper proposes three ways to combine knowledge-attention with standard self-attention:
- Multi-channel Attention (MCA): Processes data through two independent encoders and merges the features using a third attention layer.
- Softmax Interpolation (SI): Runs two separate classifiers and combines their output probabilities using a weighted average ($\beta$). This allows for high controllability; for instance, increasing the weight of the knowledge channel can improve precision.
- Knowledge-informed Self-attention (KISA): Injects the knowledge-attention directly into the internal heads of a self-attention layer.
Results and Key Insights
The proposed models achieve state-of-the-art results on the TACRED dataset, a large-scale benchmark with over 106,000 sentences. The Softmax Interpolation (SI) model achieved an F1 score of 67.8, significantly outperforming the 66.5 F1 of the standard self-attention baseline.
Key findings include:
- Complementary Strengths: Knowledge-attention acts like a high-precision rule-based system, while self-attention provides broad contextual recall. Merging them yields a "best of both worlds" performance.
- Relational Subspaces: By using a multi-head structure for knowledge-attention, the model can jointly attend to different "subspaces" of a relation, which is crucial for complex sentences where multiple indicators might be present.
- Generalization: On the smaller SemEval2010-Task8 dataset, knowledge-augmented models showed a statistically significant lead, suggesting that prior knowledge is particularly beneficial when data is limited.
Conclusion
By incorporating external lexical knowledge into the attention mechanism, the authors demonstrate that "human-in-the-loop" linguistic cues can significantly boost deep learning performance in relation extraction. While the model still faces challenges with very long-distance dependencies, future work aims to incorporate higher-level conceptual or encyclopedic knowledge from sources like Wikipedia to further enhance semantic understanding.
Summary updated