Attention-based Conditioning Methods for External Knowledge Integration

Katerina Margatina, Christos Baziotis, Alexandros Potamianos

Abstract

In this paper, we present a novel approach for incorporating external knowledge in Recurrent Neural Networks (RNNs). We propose the integration of lexicon features into the self-attention mechanism of RNN-based architectures. This form of conditioning on the attention distribution enforces the contribution of the most salient words for the task at hand. We introduce three methods, namely attentional concatenation, feature-based gating, and affine transformation. Experiments on six benchmark datasets show the effectiveness of our methods. Attentional feature-based gating yields consistent performance improvement across tasks. Our approach is implemented as a simple add-on module for RNN-based models with minimal computational overhead and can be adapted to any deep neural architecture.

1. Introduction

Modern deep learning models for Natural Language Processing (NLP) typically learn latent representations from raw text, often bypassing the rich, human-curated linguistic knowledge found in lexicons. These lexicons map words to specific traits, such as sentiment or emotion. Traditionally, this knowledge is integrated via simple concatenation at the input layer, which fails to explicitly guide the model's focus toward salient features.

We address this limitation by proposing that external knowledge should be used for conditioning—modulating the model's internal computations to better identify important words. By moving the integration of lexicons from the input layer to the attention layer, the model becomes significantly more effective at filtering noise and prioritizing task-relevant information.

2. Proposed Conditioning Methods

We developed three novel methods to integrate external lexicon features $c(w_i)$ directly into the self-attention mechanism of a Bi-LSTM architecture:

  • Attentional Concatenation: The hidden state of the LSTM is concatenated with lexicon features before calculating the attention score, adding discriminative dimensions to the word representation.
  • Attentional Feature-based Gating: This method uses a sigmoid gate derived from lexicon features to "filter" the LSTM's hidden states. It allows the model to dynamically suppress or amplify specific features based on prior knowledge.
  • Attentional Affine Transformation: Inspired by the FiLM framework, this method generates scaling ($\gamma$) and shifting ($\beta$) vectors from lexicons to transform the hidden states.
Overview of the three proposed attentional conditioning methods: (a) Attentional Concatenation, (b) Attentional Feature-Based Gating, and (c) Attentional Affine Transformation.

3. Experimental Results

Our methods were evaluated across six diverse benchmark datasets, including SST-5 (movie reviews), Sent17 (Twitter sentiment), and PhychExp (emotion recognition).

Key Findings

  • Superiority of Gating: The "Feature-based Gating" method was the standout performer, showing consistent improvements over the baseline across all datasets. Its constrained nature makes it less prone to overfitting than the more complex Affine Transformation, particularly on smaller datasets.
  • Synergistic Effects: The highest performance gains were achieved by combining lexicon features at both the embedding level (traditional concatenation) and the attention level (gating).
  • State-of-the-Art Performance: The combined "Gate + Embedding Concatenation" model achieved state-of-the-art $F_1$ scores on Sent17 (68.9) and PhychExp (57.2) at the time of publication.

4. Interpretability and Visualization

A key advantage of our approach is improved model interpretability. Visualizing the attention heatmaps reveals that conditioned models focus much more sharply on emotionally charged words. As shown in the figure below, the model's attention distribution shifts significantly depending on the target emotion, successfully identifying salient terms like "mistake" for guilt-related classifications while ignoring noise.

Attention heatmaps for the same sentence under different emotion labels. The conditioned model demonstrates the ability to shift focus to task-relevant salient words, improving both accuracy and interpretability.

5. Conclusion

Integrating external knowledge into the attention mechanism provides a computationally efficient way to boost RNN performance. These "add-on" modules require minimal overhead and offer clearer interpretability than standard black-box models. Future work will explore the integration of structured knowledge sources like Knowledge Graphs and the application of these techniques to sequence labeling tasks such as Named Entity Recognition (NER).

Summary updated

Abstract

In this paper, we present a novel approach for incorporating external knowledge in Recurrent Neural Networks (RNNs). We propose the integration of lexicon features into the self-attention mechanism of RNN-based architectures. This form of conditioning on the attention distribution, enforces the contribution of the most salient words for the task at hand. We introduce three methods, namely attentional concatenation, feature-based gating and affine transformation. Experiments on six benchmark datasets show the effectiveness of our methods. Attentional feature-based gating yields consistent performance improvement across tasks. Our approach is implemented as a simple add-on module for RNN-based models with minimal computational overhead and can be adapted to any deep neural architecture.
BibTeX
@inproceedings{margatina2019attention,
  author = {Katerina Margatina and Christos Baziotis and Alexandros Potamianos},
  title = {{Attention-based Conditioning Methods for External Knowledge Integration}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=gUps29ZSMz},
}
Figure