Importance of Self-Attention for Sentiment Analysis

Gaël Letarte, Frédérik Paradis, Philippe Giguère, François LavioletteEMNLP 2018

TL;DR

This paper introduces SANet, a novel Self-Attention Network for text classification that models word interactions without relying on recurrent or convolutional layers. SANet achieves competitive performance, particularly in sentiment analysis, and offers enhanced interpretability by visualizing attention patterns, revealing task-specific behaviors.

What Problem Does This Paper Address?

Deep neural networks, while powerful, often function as "black boxes," making it difficult to understand why they make certain predictions. This lack of interpretability hinders trust and the ability to debug or improve models. In natural language processing (NLP), attention mechanisms have emerged as a way to shed light on model decisions by highlighting which parts of the input are most relevant. However, existing attention-based methods for text classification often still rely on traditional architectures like CNNs or RNNs, or they focus on limited word interactions. This paper aims to develop a more interpretable text classification model that leverages the full power of self-attention, inspired by its success in machine translation, and to analyze how this attention mechanism behaves differently across various NLP tasks.

Methods

The core of this paper is the Self-Attention Network (SANet), an architecture inspired by the Transformer model but adapted for text classification. Unlike the Transformer's sequence-to-sequence output, SANet focuses on producing a single classification label.

Key methodological aspects include:

  • Self-Attention Mechanism: SANet utilizes self-attention, where each word in a sequence can attend to every other word, regardless of their position. This allows the model to capture complex inter-word relationships. The attention is computed using linear projections of the input embeddings as queries, keys, and values.
  • Sequence Length Agnosticism: To handle variable-length input sequences, SANet employs a global max pooling layer at the end of the network. This aggregates information across all dimensions of the sequence, allowing it to process inputs of any length (limited only by memory).
  • Residual Connections and Layer Normalization: Similar to the Transformer, SANet incorporates residual connections and layer normalization within its self-attention blocks to facilitate training of deeper networks.
  • Positional Encoding: To retain information about word order, positional encoding vectors are added to the word embeddings.
  • Task-Specific Analysis: The paper goes beyond proposing the architecture by conducting a detailed analysis of the attention patterns for two distinct tasks: topic classification and sentiment analysis. They use quantitative metrics like the Gini coefficient (for sparsity) and diagonality (for diagonal concentration) to support their qualitative observations.

Datasets

The authors evaluated SANet on seven large-scale text classification datasets, grouped into two categories:

  • Topic Classification:
    • AG's News
    • DBPedia
    • Yahoo! Answers
  • Sentiment Analysis:
    • Yelp Review Polarity
    • Yelp Review Full
    • Amazon Review Full
    • Amazon Review Polarity

For sentiment analysis, two versions were derived from the Yelp and Amazon datasets: one for predicting star ratings (1-5) and another for predicting polarity (negative/positive).

Key Insights and Ideas

  • Self-Attention is Task-Dependent: The most striking insight is that the behavior of self-attention differs significantly between topic classification and sentiment analysis. For topic classification, attention tends to be more distributed, looking for interactions between key concepts across the sequence. In contrast, for sentiment analysis, attention is more localized, focusing on neighboring words and skip-bigrams.
  • Interpretability Reveals Model Behavior: Visualizing the attention matrices provides a clear window into how SANet processes text. This interpretability is crucial for understanding why the model performs well (or poorly) on specific tasks.
  • Neighboring Word Interactions are Key for Sentiment: The strong diagonal band pattern observed in sentiment analysis attention matrices suggests that sentiment is often derived from the immediate context and relationships between adjacent or closely spaced words. This aligns with previous findings on the importance of word bigrams for sentiment analysis.
  • Global Max Pooling is Effective for Length Agnosticism: The use of global max pooling allows SANet to handle sequences of varying lengths without truncation or padding, a common issue with CNN-based approaches.
  • Yelp Polarity Anomaly: The paper highlights an interesting anomaly where SANet performs poorly on Yelp Review Polarity, despite performing well on Yelp Review Full. This suggests that the specific class merging or dataset characteristics might pose a challenge for the self-attention mechanism in this particular case.

Key Results

  • Performance: SANet achieved competitive test error rates across various datasets, often performing comparably to or better than state-of-the-art models, especially in sentiment analysis. For instance, SANet (big) achieved an error rate of 5.55% on Amazon Review Full, which is very close to the state-of-the-art.
  • Sentiment Analysis Gains: Self-attention consistently improved accuracy for sentiment analysis tasks, showing gains of around 2% compared to a baseline without attention.
  • Topic Classification No Gain: For topic classification, self-attention did not yield significant performance improvements over the baseline.
  • Quantitative Analysis:
    • Gini Coefficient: Sentiment analysis datasets showed a higher Gini coefficient (87.15% on average, excluding Yelp Polarity) compared to topic classification (63.57%), indicating sparser attention distributions in sentiment analysis.
  • Diagonality: For a bandwidth of 3, sentiment analysis attention matrices had 63.48% of weights within the band, while topic classification had only 16.12%. This quantitatively confirms the localized, diagonal attention pattern in sentiment analysis.

Limitations and Future Work

  • Yelp Review Polarity Anomaly: The paper acknowledges the unexplained poor performance and inadequate attention behavior on the Yelp Review Polarity dataset. Future work could investigate the reasons for this discrepancy, especially since Yelp Review Polarity is a subset of Yelp Review Full.
  • Linguistic Investigation: A deeper linguistic analysis of the impact of closely interacting words in sentiment analysis is suggested.
  • Broader Applications: The authors propose applying SANet to other datasets and tasks, such as bullying detection on social media and genomic data analysis.
  • Complementary Interpretability: They suggest exploring the combination of global max pooling with other interpretability tools like Class Activation Mapping (CAM) for a more comprehensive understanding of word-level context.
  • Focus on Pairs vs. Individual Words: The current SANet focuses on interactions between word pairs. Future work could explore mechanisms that allow attention on individual words while still considering their context, potentially offering a complementary view to the current approach.

Summary updated

BibTeX
@inproceedings{letarte2018importance,
  author = {Gaël Letarte and Frédérik Paradis and Philippe Giguère and François Laviolette},
  title = {{Importance of Self-Attention for Sentiment Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure