Abstract
In this paper, we investigate the effect of different hyperparameters as well as different combinations of hyperparameters settings on the performance of the Attention-Gated Convolutional Neural Networks (AGCNNs), e.g., the kernel window size, the number of feature maps, the keep rate of the dropout layer, and the activation function. We draw practical advice from a wide range of empirical results. Through the sensitivity analysis, we further improve the hyperparameters settings of AGCNNs. Experiments show that our proposals could achieve an average of 0.81% and 0.67% improvements on AGCNN-NLReLU-rand and AGCNN-SELU-rand, respectively; and an average of 0.47% and 0.45% improvements on AGCNN-NLReLU-static and AGCNN-SELU-static, respectively.
Introduction
Convolutional Neural Networks (CNNs) have become a cornerstone for Natural Language Processing (NLP) tasks, particularly sentence classification. The Attention-Gated variant (AGCNN) enhances standard CNNs by incorporating an internal gating mechanism to filter significant features. However, AGCNNs introduce a significant "hyperparameter tax," as they are notoriously sensitive to configuration settings. Given the high computational costs of exhaustive searches, this study provides a systematic sensitivity analysis to establish empirical "rules of thumb" for practitioners.
Methodology
The research utilizes a "one-at-a-time" sensitivity analysis based on a baseline AGCNN-static model. The study isolates and varies several key components:
- Kernel Window Sizes: Testing permutations for both the first convolutional layer and the attention-gated layer.
- Feature Map Counts: Evaluating model capacity versus GPU memory constraints.
- Dropout Regularization: Identifying optimal keep rates.
- Activation Functions: Comparing ReLU, SELU (Scaled Exponential Linear Unit), and NLReLU (Natural Logarithm rescaled ReLU).
Performance was evaluated across six benchmarks: CR (Customer Reviews), MR (Movie Reviews), Subj (Subjectivity), SST-1/SST-2 (Stanford Sentiment Treebank), and TREC (Question Classification), using 10-fold cross-validation.
Hyperparameter Sensitivity Analysis
Convolutional Kernel Sizes
A critical finding is the effectiveness of "multi-scale" kernels. Rather than a single optimal size, combining consecutive sizes (e.g., 1 through 5) in the first layer allows the model to capture diverse semantic granularities. As shown in Figure 4, performance varies significantly across datasets and activation functions as the kernel size changes.
Attention Layer Symmetry
For the attention-gated layer, odd-numbered kernel sizes (1, 3, 5, 7) generally outperform even ones. This symmetry provides a balanced context around target features, leading to more precise attention weights. Figure 5 illustrates the accuracy fluctuations relative to the attention layer's kernel window size.
Activation and Feature Maps
While SELU often yields the highest peak accuracy, it increases the model's sensitivity to other hyperparameter shifts. In contrast, ReLU offers greater stability, particularly for specific tasks like question classification. Regarding feature maps, increasing the count beyond 400 in the first layer provides diminishing returns while consuming excessive memory.
Empirical Results
By adopting a (1,2,3,4,5) kernel mix for the first layer and a (1,3,5,7) mix for the attention layer, the proposed configurations consistently outperformed baseline AGCNN implementations. Average improvements reached 0.81% for AGCNN-rand (NLReLU) and 0.47% for AGCNN-static (NLReLU). Notable gains were observed in the TREC dataset, where accuracy rose from ~93.5% to over 95.2%.
Conclusion
This sensitivity analysis demystifies the tuning process for AGCNNs, offering a practical roadmap for optimization. While dropout rates remain dataset-dependent, the identified "rules of thumb" for kernel sizing and activation choice provide a robust starting point for high-performance sentence classification. Future research will explore these architectural sensitivities in the context of transformer-based embeddings.
Summary updated