Sentiment Analysis Using Multi-Head Attention Capsules With Multi-Channel CNN and Bidirectional GRU

Yan Cheng, Huan Sun, Haomai Chen, Meng Li, Yingying Cai, Zhuang Cai, Jing HuangIEEE Access 2021 · DOI

Abstract

Existing text sentiment analysis methods mostly rely on a large number of language knowledge and sentiment resources. This paper proposes the Multi-channel convolution and bidirectional GRU multi-head attention capsule (AT-MC-BiGRU-Capsule), which uses vector neurons to replace scalar neurons to model text emotions. Traditional methods often fail to extract multi-level features of text sequences. By combining Multi-head attention, Convolutional Neural Networks (CNN), and Bidirectional Gated Recurrent Units (Bi-GRU), the proposed model captures local features, global semantic features, and word dependencies. Experiments on three English and one Chinese dataset demonstrate that this approach outperforms several baseline models.

Introduction

Sentiment analysis is a critical task in natural language processing (NLP), yet it faces significant hurdles. Dictionary-based methods require extensive manual maintenance and struggle with evolving language. Meanwhile, standard deep learning models often fail to capture the multi-level nature of text, missing either local n-gram features or global context.

The researchers developed the AT-MC-BiGRU-Capsule model to address these gaps. By replacing traditional scalar neurons with "capsules" (vector neurons), the model preserves complex relationships between words and their emotional weights, providing a more nuanced understanding of sentiment.

Proposed Architecture

The AT-MC-BiGRU-Capsule architecture integrates several advanced deep learning components to process text comprehensively, as shown in the overall framework below.

The overall architecture of the AT-MC-BiGRU-Capsule model, showing the flow from word embeddings to emotional capsules.

1. Attention Layer

The model begins with a Multi-Head Attention mechanism. This layer encodes dependencies between words, allowing the model to focus on sentiment-bearing words (e.g., "excellent" or "disappointing") regardless of their position in the sentence.

Detailed view of the Multi-Head Attention mechanism, utilizing scaled dot-product attention across multiple heads.

2. Feature Extraction Layer

A hybrid approach is used for feature extraction:

  • Multi-Channel CNN: Extracts local n-gram features, capturing short-range patterns and phrases.
  • Bidirectional GRU (Bi-GRU): Processes text in both forward and backward directions to capture long-range semantic dependencies and global context.

3. Feature Fusion and Capsule Layer

Local and global features are fused and passed through a Global Average Pooling layer to prevent overfitting. The core innovation lies in the Emotional Capsule Construction Layer. Unlike scalar neurons that output a single probability, capsules use vector neurons to represent the "state" of an emotion, including its intensity and orientation. If the activation probability of a specific capsule is the highest, the model classifies the text accordingly.

Experimental Results

The model was evaluated on four benchmarks: MR (short movie reviews), IMDB (long reviews), SST-5 (fine-grained sentiment), and COAE2014 (Chinese opinions).

  • Performance: The model achieved 85.3% accuracy on MR and 91.5% on IMDB. On the challenging SST-5 dataset, it reached 50.0%, significantly outperforming standard CNNs (46.9%).
  • BERT Integration: When paired with BERT dynamic word vectors, accuracy on IMDB improved to 92.7%, surpassing SentiBERT and other transformer-based fine-tuning methods.
  • Cross-Linguistic Robustness: Strong performance on the Chinese COAE dataset confirms the model's effectiveness across different languages.

Conclusion

The AT-MC-BiGRU-Capsule model demonstrates the power of combining local and global feature extraction with vector-based capsule networks. By moving beyond scalar representations and utilizing multi-head attention, the model achieves state-of-the-art results in sentiment classification. Future work will focus on optimizing the internal routing of capsules and improving stability for informal social media text.

Summary updated

Abstract

Existing text sentiment analysis methods mostly rely on a large number of language knowledge and sentiment resources. This paper proposes the Multi-channel convolution and bidirectional GRU multi-head attention capsule (AT-MC-BiGRU-Capsule), which uses vector neurons to replace scalar neurons to model text emotions, and uses capsules to characterize text emotions. In addition, traditional methods cannot extract the multi-level features of text sequence well. Multi-head attention can encode the dependencies between words, capture sentiment words in text, and using Convolutional Neural Network (CNN) and Bidirectional gated recurrent unit network (Bi-GRU) to extract local features and global semantic features of text respectively, the global average pooling layer is introduced to obtain the multi-level feature representation of the text sequence more comprehensively. This paper selects three English datasets and one Chinese dataset in the general corpus of sentiment classification to conduct experiments, and achieves better results than other baseline models.
BibTeX
@inproceedings{cheng2021sentiment,
  author = {Yan Cheng and Huan Sun and Haomai Chen and Meng Li and Yingying Cai and Zhuang Cai and Jing Huang},
  title = {{Sentiment Analysis Using Multi-Head Attention Capsules With Multi-Channel CNN and Bidirectional GRU}},
  booktitle = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3073988},
}
Figure