Self-Attention: A Better Building Block for Sentiment Analysis Neural Network Classifiers

Artaches Ambartsoumian, Fred PopowichEMNLP 2018

TL;DR

This paper investigates the effectiveness of self-attention networks (SANs) as a core building block for sentiment analysis classifiers. The authors demonstrate that SANs, particularly when combined with relative positional representations, outperform traditional recurrent (LSTM, BiLSTM) and convolutional (CNN) neural networks across multiple sentiment analysis datasets, offering a more efficient and accurate approach.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone or opinion expressed in text, has seen significant advancements with the rise of neural networks. While Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been dominant, they often struggle with capturing long-range dependencies in text and can be computationally expensive. This paper addresses the need for more effective and efficient neural architectures for sentiment analysis by exploring self-attention networks (SANs), a relatively newer approach that eschews recurrence and convolutions, and examining their suitability as a fundamental component for sentiment classification.

Methods

The core of this research lies in evaluating Self-Attention Networks (SANs) for sentiment analysis. The authors propose a Simple Self-Attention (SSAN) model, which uses self-attention as its primary mechanism. They explore both single-layer and two-layer SSAN architectures. A key aspect of their methodology is the investigation of different techniques for incorporating positional information into these attention mechanisms, as self-attention itself is inherently order-invariant. They compare:

  • Sinusoidal Positional Encoding (PE): A standard method where fixed sine and cosine functions are added to input embeddings.
  • Learned Position Encoding: Where unique, trainable vectors are added to input embeddings for each position.
  • Relative Position Representations (RPR): A novel approach where the attention mechanism explicitly learns the relative positional information between sequence elements. This method was found to be particularly effective.

They also compare their SSAN models against established baselines like LSTM, BiLSTM, and CNN, using the same word embeddings and preprocessing pipelines for a fair comparison. Furthermore, they evaluate a more complex Transformer encoder architecture to understand the benefits of multi-head attention.

Datasets

The paper evaluates their proposed models on six publicly available sentiment analysis datasets:

  • Stanford Sentiment Treebank (SST-fine): A fine-grained dataset with five sentiment classes (very-negative to very-positive) derived from movie reviews.
  • SST-binary: A binary version of SST-fine, collapsing negative and positive classes.
  • OpeNER: A dataset of hotel reviews with four sentiment classes.
  • SenTube-A & SenTube-T: Datasets consisting of YouTube comments with two sentiment classes (positive/negative).
  • SemEval Twitter dataset (SemEval): A dataset of tweets with three sentiment classes (positive, negative, neutral).

These datasets were chosen for their varied properties and are standard benchmarks in sentiment analysis research.

Key Insights and Ideas

  • Self-Attention as a Superior Building Block: The research strongly suggests that self-attention is a more effective fundamental component for sentiment analysis than traditional recurrence or convolutions, leading to higher accuracy.
  • Relative Positional Representations Shine: Explicitly modeling relative positional information (RPR) significantly boosts the performance of self-attention models compared to using positional encodings or no positional information at all. This highlights the importance of sequence order in sentiment analysis.
  • Simplicity Can Be Powerful: A single layer of self-attention with RPR (1-Layer-SSAN+RPR) often outperforms more complex architectures, including multi-layer SSANs and even the Transformer encoder, on these datasets. This suggests that for many sentiment analysis tasks, a simpler SAN architecture is sufficient and more efficient.
  • Efficiency Gains: Self-attention models, despite achieving better accuracy, can be more efficient in terms of trainable parameters and training time compared to RNNs.

Key Results

  • Accuracy Improvements: The proposed SSAN models, especially with RPR, consistently achieved higher Macro-Average accuracy (average performance across all datasets) than LSTM, BiLSTM, and CNN baselines. For instance, 1-Layer-SSAN+RPR achieved top scores on five out of six datasets.
  • RPR vs. PE: Using RPR for 1-Layer-SSAN increased Macro-Avg accuracy by 2.8% compared to positional encodings (PE) and 0.9% compared to no positional information.
  • Efficiency: SSAN models had fewer trainable parameters and shorter training times than comparable RNN models. For example, SSAN achieved higher accuracy than LSTMs with 35% fewer parameters and a 3.5x shorter training time.
  • Inference Time: While SANs were faster to train, their inference times were slightly slower than LSTMs but faster than BiLSTMs.
  • SenTube Dataset Anomaly: The SenTube datasets showed lower accuracy despite having binary classes, which the authors attribute to their length and low word embedding coverage. However, SSANs still performed relatively well, indicating their strength in capturing long-range dependencies.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the tested datasets might be relatively small, which could explain why a single self-attention layer was sufficient and why positional encodings performed less optimally than RPR or no positional information. They hypothesize that for larger datasets, stacked multi-head self-attention architectures might yield even better results.
  • Parameter Tuning: The training process was simplified by fixing many hyperparameters and only tuning the learning rate. The authors note that for smaller input embedding dimensions, individual tuning of learning rates might be necessary for optimal performance.
  • Future Work: The authors plan to extend their SSAN models to achieve state-of-the-art results on the complete SST dataset and explore their performance on much larger datasets. They anticipate that stacked multi-head self-attention architectures will be more effective on larger datasets while remaining efficient.

Summary updated

BibTeX
@inproceedings{ambartsoumian2018self,
  author = {Artaches Ambartsoumian and Fred Popowich},
  title = {{Self-Attention: A Better Building Block for Sentiment Analysis Neural Network Classifiers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure