A Sensitivity Analysis of Attention-Gated Convolutional Neural Networks for Sentence Classification

Yang Liu, Jianpeng Zhang, Chao Gao, Qu Jing-hua, Lixin Ji

Abstract

In this paper, we investigate the effect of different hyperparameters as well as different combinations of hyperparameters settings on the performance of the Attention-Gated Convolutional Neural Networks (AGCNNs), e.g., the kernel window size, the number of feature maps, the keep rate of the dropout layer, and the activation function. We draw practical advice from a wide range of empirical results. Through the sensitivity analysis, we further improve the hyperparameters settings of AGCNNs. Experiments show that our proposals could achieve an average of 0.81% and 0.67% improvements on AGCNN-NLReLU-rand and AGCNN-SELU-rand, respectively; and an average of 0.47% and 0.45% improvements on AGCNN-NLReLU-static and AGCNN-SELU-static, respectively.

Introduction

Convolutional Neural Networks (CNNs) have become a cornerstone for Natural Language Processing (NLP) tasks, particularly sentence classification. The Attention-Gated variant (AGCNN) enhances standard CNNs by incorporating an internal gating mechanism to filter significant features. However, AGCNNs introduce a significant "hyperparameter tax," as they are notoriously sensitive to configuration settings. Given the high computational costs of exhaustive searches, this study provides a systematic sensitivity analysis to establish empirical "rules of thumb" for practitioners.

Methodology

The research utilizes a "one-at-a-time" sensitivity analysis based on a baseline AGCNN-static model. The study isolates and varies several key components:

  • Kernel Window Sizes: Testing permutations for both the first convolutional layer and the attention-gated layer.
  • Feature Map Counts: Evaluating model capacity versus GPU memory constraints.
  • Dropout Regularization: Identifying optimal keep rates.
  • Activation Functions: Comparing ReLU, SELU (Scaled Exponential Linear Unit), and NLReLU (Natural Logarithm rescaled ReLU).

Performance was evaluated across six benchmarks: CR (Customer Reviews), MR (Movie Reviews), Subj (Subjectivity), SST-1/SST-2 (Stanford Sentiment Treebank), and TREC (Question Classification), using 10-fold cross-validation.

Hyperparameter Sensitivity Analysis

Convolutional Kernel Sizes

A critical finding is the effectiveness of "multi-scale" kernels. Rather than a single optimal size, combining consecutive sizes (e.g., 1 through 5) in the first layer allows the model to capture diverse semantic granularities. As shown in Figure 4, performance varies significantly across datasets and activation functions as the kernel size changes.

Sensitivity of accuracy to the kernel window size of the first convolutional layer using NLReLU (a) and SELU (b) activation functions.

Attention Layer Symmetry

For the attention-gated layer, odd-numbered kernel sizes (1, 3, 5, 7) generally outperform even ones. This symmetry provides a balanced context around target features, leading to more precise attention weights. Figure 5 illustrates the accuracy fluctuations relative to the attention layer's kernel window size.

Impact of kernel window size in the attention-gated layer on model accuracy across different datasets.

Activation and Feature Maps

While SELU often yields the highest peak accuracy, it increases the model's sensitivity to other hyperparameter shifts. In contrast, ReLU offers greater stability, particularly for specific tasks like question classification. Regarding feature maps, increasing the count beyond 400 in the first layer provides diminishing returns while consuming excessive memory.

Empirical Results

By adopting a (1,2,3,4,5) kernel mix for the first layer and a (1,3,5,7) mix for the attention layer, the proposed configurations consistently outperformed baseline AGCNN implementations. Average improvements reached 0.81% for AGCNN-rand (NLReLU) and 0.47% for AGCNN-static (NLReLU). Notable gains were observed in the TREC dataset, where accuracy rose from ~93.5% to over 95.2%.

Conclusion

This sensitivity analysis demystifies the tuning process for AGCNNs, offering a practical roadmap for optimization. While dropout rates remain dataset-dependent, the identified "rules of thumb" for kernel sizing and activation choice provide a robust starting point for high-performance sentence classification. Future research will explore these architectural sensitivities in the context of transformer-based embeddings.

Summary updated

Abstract

In this paper, we investigate the effect of different hyperparameters as well as different combinations of hyperparameters settings on the performance of the Attention-Gated Convolutional Neural Networks (AGCNNs), e.g., the kernel window size, the number of feature maps, the keep rate of the dropout layer, and the activation function. We draw practical advice from a wide range of empirical results. Through the sensitivity analysis, we further improve the hyperparameters settings of AGCNNs. Experiments show that our proposals could achieve an average of 0.81% and 0.67% improvements on AGCNN-NLReLU-rand and AGCNN-SELU-rand, respectively; and an average of 0.47% and 0.45% improvements on AGCNN-NLReLU-static and AGCNN-SELU-static, respectively.
BibTeX
@misc{liu2019sensitivity,
  author = {Yang Liu and Jianpeng Zhang and Chao Gao and QU Jing-hua and Lixin Ji},
  title = {{A Sensitivity Analysis of Attention-Gated Convolutional Neural Networks for Sentence Classification}},
  year = {2019},
  eprint = {1908.06263},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.06263},
}
Figure