Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores

Dongjun Jang, Jinwoong Kim, Hyopil ShinNAACL 2024

TL;DR

This paper introduces "Sentiment Lexical Attention," a novel method to inject sentiment knowledge directly into Transformer models' self-attention mechanism. By fusing this sentiment-derived attention with the original attention scores, they achieve state-of-the-art performance on Korean sentiment classification and demonstrate improved robustness on out-of-domain tasks.

What Problem Does This Paper Address?

Pre-trained language models (PLMs) like BERT have revolutionized Natural Language Processing (NLP), largely due to their powerful self-attention mechanisms that capture relationships between words. However, a key question remains: can we further enhance these models by explicitly injecting external knowledge, specifically sentiment-related lexical features, into the self-attention process? While previous work has explored knowledge injection, this paper specifically targets improving sentiment analysis by leveraging semantic-polarity scores to guide attention, aiming to create a more nuanced understanding of sentiment-laden text.

Methods

The core innovation is Sentiment Lexical Attention (SLA). This is derived from semantic-polarity scores of words and phrases within a sentence. The authors propose a method to quantify the semantic-polarity similarity between token pairs, creating a matrix that highlights sentiment linkages. This SLA matrix is then fused with the original attention scores.

Instead of using a Hadamard product (which can be sensitive to negative values and potentially distort relationships) as in prior work (Xia et al., 2021), this paper proposes a linear combination (summation) of the SLA and the original attention scores. This "Sentiment-Fused Attention" is integrated into the self-attention mechanism before the softmax activation. The authors also explore different configurations for injecting this fused attention, including applying it to all attention heads across all layers or focusing it on the final attention head.

Datasets

  • NSMC (Naver Sentiment Movie Corpus): A benchmark dataset for Korean sentiment classification, used for training and evaluating the primary sentiment analysis task. It consists of 200K reviews (150K for training, 50K for testing).
  • CARBD-ko dataset: A Korean dataset used for extracting context polarity vectors to initialize the Sentiment Lexical Attention.
  • Out-of-Domain Datasets:
    • KorNLI
    • PAWS-ko
    • Hate Speech Detection
    • Question-Pair benchmark

Key Insights and Ideas

  • Sentiment as a Direct Attention Guide: The paper demonstrates that sentiment information, when directly injected into the attention mechanism, can significantly boost performance, especially for sentiment-related tasks.
  • Summation over Hadamard Product: The choice of a linear combination (summation) for fusing SLA with original attention scores proves more stable and effective than the Hadamard product, avoiding potential distortion of token relationships.
  • CLS Token's Role: Analysis of the CLS token's attention dynamics reveals that while the CLS token itself might not be directly influenced by SLA in terms of its own context polarity, the changes in other tokens' attention distributions towards the CLS token are a key driver of performance improvements.
  • Task and Model Dependency: While the approach shows promise, the effectiveness of knowledge injection can vary across different models and tasks, highlighting the need for careful consideration of the target domain and architecture.

Key Results

  • State-of-the-Art on NSMC: The proposed Sentiment-Fused Attention achieved state-of-the-art (SoTA) performance on the NSMC benchmark for Korean sentiment classification, with accuracy improvements of up to 0.68% over previous methods.
  • Out-of-Domain Robustness: The method showed consistent performance enhancements across various out-of-domain tasks, indicating its generalizability. For instance, on the Question-Pair task, one model achieved a significant improvement of +2.17%.
  • Attlast+√dk Configuration: The scenario where Sentiment Lexical Attention was injected only into the final attention head (Attlast) and scaled by √dk demonstrated the most significant performance improvements and the smallest standard deviation across models, suggesting a stable and effective configuration.

Limitations and Future Work

  • Task Relevance Assumption: The current approach assumes a direct relevance of semantic-polarity scores to the task. This limits its applicability to tasks where such direct relevance isn't clear, requiring careful selection of knowledge for new domains.
  • Model Dependency: Performance gains can be model-dependent. The authors suggest preliminary tests to assess compatibility with different Transformer architectures.
  • Sentiment Lexicon Quality: The effectiveness heavily relies on the quality and accuracy of the sentiment lexicon used. Inaccuracies can lead to significant performance deviations.
  • Cross-Lingual Applicability: While the core principle is language-agnostic, adapting it to other languages requires careful curation of sentiment lexicons and consideration of linguistic nuances. Future work will explore this cross-linguistic applicability.
  • Generalizing Lexical Features: The authors suggest exploring domain-specific knowledge injection and integrating multiple types of lexical knowledge to enhance versatility across a wider range of tasks beyond sentiment analysis.
  • Understanding Dynamics: Further research is needed to fully understand the precise influence of injected knowledge on the model's training and decision-making processes, especially in complex real-world applications.

Summary updated

BibTeX
@inproceedings{jang2024enhancing,
  author = {Dongjun Jang and Jinwoong Kim and Hyopil Shin},
  title = {{Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure