Abstract
Existing text sentiment analysis methods mostly rely on a large number of language knowledge and sentiment resources. This paper proposes the Multi-channel convolution and bidirectional GRU multi-head attention capsule (AT-MC-BiGRU-Capsule), which uses vector neurons to replace scalar neurons to model text emotions. Traditional methods often fail to extract multi-level features of text sequences. By combining Multi-head attention, Convolutional Neural Networks (CNN), and Bidirectional Gated Recurrent Units (Bi-GRU), the proposed model captures local features, global semantic features, and word dependencies. Experiments on three English and one Chinese dataset demonstrate that this approach outperforms several baseline models.
Introduction
Sentiment analysis is a critical task in natural language processing (NLP), yet it faces significant hurdles. Dictionary-based methods require extensive manual maintenance and struggle with evolving language. Meanwhile, standard deep learning models often fail to capture the multi-level nature of text, missing either local n-gram features or global context.
The researchers developed the AT-MC-BiGRU-Capsule model to address these gaps. By replacing traditional scalar neurons with "capsules" (vector neurons), the model preserves complex relationships between words and their emotional weights, providing a more nuanced understanding of sentiment.
Proposed Architecture
The AT-MC-BiGRU-Capsule architecture integrates several advanced deep learning components to process text comprehensively, as shown in the overall framework below.
1. Attention Layer
The model begins with a Multi-Head Attention mechanism. This layer encodes dependencies between words, allowing the model to focus on sentiment-bearing words (e.g., "excellent" or "disappointing") regardless of their position in the sentence.
2. Feature Extraction Layer
A hybrid approach is used for feature extraction:
- Multi-Channel CNN: Extracts local n-gram features, capturing short-range patterns and phrases.
- Bidirectional GRU (Bi-GRU): Processes text in both forward and backward directions to capture long-range semantic dependencies and global context.
3. Feature Fusion and Capsule Layer
Local and global features are fused and passed through a Global Average Pooling layer to prevent overfitting. The core innovation lies in the Emotional Capsule Construction Layer. Unlike scalar neurons that output a single probability, capsules use vector neurons to represent the "state" of an emotion, including its intensity and orientation. If the activation probability of a specific capsule is the highest, the model classifies the text accordingly.
Experimental Results
The model was evaluated on four benchmarks: MR (short movie reviews), IMDB (long reviews), SST-5 (fine-grained sentiment), and COAE2014 (Chinese opinions).
- Performance: The model achieved 85.3% accuracy on MR and 91.5% on IMDB. On the challenging SST-5 dataset, it reached 50.0%, significantly outperforming standard CNNs (46.9%).
- BERT Integration: When paired with BERT dynamic word vectors, accuracy on IMDB improved to 92.7%, surpassing SentiBERT and other transformer-based fine-tuning methods.
- Cross-Linguistic Robustness: Strong performance on the Chinese COAE dataset confirms the model's effectiveness across different languages.
Conclusion
The AT-MC-BiGRU-Capsule model demonstrates the power of combining local and global feature extraction with vector-based capsule networks. By moving beyond scalar representations and utilizing multi-head attention, the model achieves state-of-the-art results in sentiment classification. Future work will focus on optimizing the internal routing of capsules and improving stability for informal social media text.
Summary updated