Abstract
In this paper, we present a novel approach for incorporating external knowledge in Recurrent Neural Networks (RNNs). We propose the integration of lexicon features into the self-attention mechanism of RNN-based architectures. This form of conditioning on the attention distribution enforces the contribution of the most salient words for the task at hand. We introduce three methods: attentional concatenation, feature-based gating, and affine transformation. Experiments on six benchmark datasets show the effectiveness of our methods. Attentional feature-based gating yields consistent performance improvement across tasks. Our approach is implemented as a simple add-on module for RNN-based models with minimal computational overhead and can be adapted to any deep neural architecture.
Introduction
While modern deep learning models are excellent at learning latent patterns from raw data, they often ignore "prior knowledge"—the decades of work linguists have put into building lexicons that categorize words by sentiment, emotion, or psychological properties. The standard way to use this knowledge is to concatenate lexicon features to the word embeddings at the input layer. However, this is often inefficient; in long sentences, the model might still struggle to "focus" on the most important words for a specific task. We propose that external knowledge should directly guide the model’s attention toward the most informative parts of the text.
Proposed Methodology
We introduce three novel methods to integrate lexicon features $c(w_i)$ into the self-attention mechanism of an LSTM-based architecture:
- Attentional Concatenation: The hidden state of the LSTM and the lexicon features are joined together before being passed through a non-linear layer to calculate attention scores, making learned representations more discriminative.
- Attentional Feature-based Gating: The model uses lexicon features to create a "mask" (a vector of values between 0 and 1). This mask is multiplied by the LSTM's hidden state, effectively filtering out irrelevant information and highlighting salient dimensions based on the lexicon data.
- Attentional Affine Transformation: Inspired by the FiLM (Feature-wise Linear Modulation) framework, this method uses lexicon features to conditionally scale and shift the hidden states.
We also evaluate a "hybrid" approach that combines Feature-based Gating in the attention layer with traditional Embedding Concatenation at the input layer.
Experimental Setup
The methods were evaluated across six diverse Natural Language Processing (NLP) benchmarks, including SST-5 (fine-grained sentiment), Sent17 (Twitter sentiment), and PsychExp (emotion recognition). We utilized a suite of well-known lexicons for external knowledge, such as LIWC (psycho-linguistic), AFINN (sentiment), and Emolex (emotion).
Key Results and Insights
Our experiments demonstrate that directly conditioning the attention mechanism is more effective than input-level concatenation alone. The Gating mechanism consistently performed the best, acting as a powerful feature selector that allows the model to ignore noise in hidden states that doesn't align with external linguistic evidence.
At the time of publication, the "gate + emb. conc." hybrid model achieved state-of-the-art (SOTA) performance on the Sent17 and PsychExp datasets. Furthermore, the approach is highly efficient, implemented as a simple add-on module with minimal extra computational cost.
Interpretability
By analyzing attention heatmaps, we found that our method forces the model to focus on task-relevant words that a baseline model might overlook. For example, in an emotion recognition task, the gated model correctly identifies words like "mistake" or "occurred" as highly salient, whereas the baseline distributes attention more broadly.
Conclusion and Future Work
This work shows that attention is an ideal location for integrating external knowledge. While the Affine Transformation method showed promise, it likely requires larger datasets to avoid overfitting due to its higher capacity. Future research will explore integrating more complex structures, such as full Knowledge Bases or graphs, and testing these methods on sequence labeling tasks like Named Entity Recognition (NER).
Summary updated