Abstract
The classification of sentences is very challenging, since sentences contain the limited contextual information. In this paper, we proposed an Attention-Gated Convolutional Neural Network (AGCNN) for sentence classification, which generates attention weights from the feature's context windows of different sizes by using specialized convolution encoders. It makes full use of limited contextual information to extract and enhance the influence of important features in predicting the sentence's category. Experimental results demonstrated that our model can achieve up to 3.1% higher accuracy than standard CNN models, and gain competitive results over the baselines on four out of the six tasks. Besides, we designed an activation function, namely, Natural Logarithm rescaled Rectified Linear Unit (NLReLU). Experiments showed that NLReLU can outperform ReLU and is comparable to other well-known activation functions on AGCNN.
Introduction
Sentence classification—the task of assigning categories to short text snippets like movie reviews or technical queries—is a cornerstone of natural language processing. However, it remains deceptively difficult because sentences lack the statistical redundancy and rich contextual cues found in longer documents. While Convolutional Neural Networks (CNNs) have become a standard tool for this task due to their efficiency in capturing local n-gram patterns, they often operate as "black boxes." Specifically, standard CNNs rely on simple pooling layers to select features, often failing to distinguish between critical sentiment-bearing words and neutral linguistic noise.
To address this, researchers have presented the Attention-Gated Convolutional Neural Network (AGCNN). This architecture introduces a mechanism to weight feature importance based on surrounding context before the pooling stage, paired with a novel activation function, NLReLU, designed to stabilize network activations.
The AGCNN Architecture
The primary innovation of AGCNN is the insertion of an Attention-Gated Layer between the initial convolution and the final pooling layer. Unlike standard models where feature maps are passed directly to pooling, AGCNN employs specialized "context encoders."
As shown in the architecture diagram, the model uses convolution kernels of varying window sizes (typically $k=1, 3, 5$) to scan the feature maps. These encoders generate an attention weight matrix that is multiplied element-wise with the original features. This process allows the model to "turn up the volume" on features that are deemed significant by their context while muting irrelevant ones. By using multiple window sizes, the model captures multi-grained attention, identifying importance at the level of individual words as well as short phrases.
NLReLU: A New Activation Function
Deep networks often struggle with "exploding" activations when using standard functions like ReLU. The authors propose Natural Logarithm rescaled Rectified Linear Unit (NLReLU), defined as:
$$f(x) = \ln(x + 1.0) \text{ for } x > 0, \text{ else } 0$$
By applying a logarithmic transformation to positive activations, NLReLU squashes extreme outliers and reduces heteroscedasticity—the uneven variance of data across layers. This ensures that the mean and variance of neuron activations remain stable, facilitating more robust learning and sensitivity to small-value activations.
Experimental Results and Insights
The model was evaluated across six benchmark datasets, including movie reviews (MR, SST), customer reviews (CR), and question classification (TREC).
Performance Gains
AGCNN consistently outperformed standard CNN baselines, with accuracy improvements ranging from 0.4% to 3.1%. Notably, the model achieved state-of-the-art or highly competitive results on four of the six tasks. On the TREC dataset, AGCNN even surpassed traditional SVM-based methods, which are historically strong in that domain.
Visualizing Attention as a Noise Filter
The effectiveness of the gating mechanism is best understood through visualization. Heat-maps of the attention weights reveal that while standard CNNs are often distracted by neutral words (e.g., "a", "the", "history"), AGCNN successfully suppresses these and highlights sentiment-heavy terms.
As illustrated in the heatmaps, the gating mechanism acts as a sophisticated noise filter, ensuring that only the most relevant contextual information reaches the final classification stage. This multi-grained approach allows the network to focus on the words that actually drive sentiment and meaning.
Conclusion
The AGCNN offers a powerful yet efficient way to enhance sentence classification by explicitly modeling feature importance through context. By combining multi-grained attention gates with the stabilizing effects of the NLReLU activation function, the model achieves significant performance boosts over standard architectures. Future work may involve integrating these attention mechanisms into tree-structured networks to better capture complex syntactic relationships.
Summary updated