Introduction
This research demystifies the "black art" of hyperparameter tuning for one-layer Convolutional Neural Networks (CNNs) in the context of sentence classification. While CNNs have become a standard baseline for NLP tasks like sentiment analysis, they come with a dizzying array of "knobs" to turn: filter widths, number of feature maps, activation functions, pooling strategies, and regularization terms. For practitioners, exploring this vast space is computationally expensive and often feels like guesswork.
By conducting a massive sensitivity analysis across nine datasets, the authors aim to distinguish between important and inconsequential design decisions. They also highlight a major pitfall in machine learning research: reporting only mean accuracy. Because training is stochastic (due to random initialization and dropout), the same model on the same data can produce significantly different results across runs. This paper quantifies that variance to ensure findings are robust.
Methods
The authors focus on a simple, one-layer CNN architecture popularized by Kim (2014). The process involves establishing a "standard" baseline configuration (ReLU activation, 1-max pooling, 100 feature maps per filter size) and systematically varying one hyperparameter at a time while holding others constant.
To account for stochasticity, instead of running an experiment once, they replicate 10-fold cross-validation up to 100 times per configuration to report the mean, minimum, and maximum performance. Finally, they compare CNN performance against traditional baselines like Linear SVMs and Logistic Regression using both N-grams and averaged word vectors.
Datasets
The study uses nine diverse sentence classification datasets:
- Sentiment/Polarity: MR (Movie Reviews), SST-1 & SST-2 (Stanford Sentiment Treebank), CR (Customer Reviews), MPQA (Opinion polarity).
- Classification: Subj (Subjectivity), TREC (Question types).
- Challenging/Imbalanced: Opi (Opinosis - user reviews), Irony (Reddit comments labeled for irony).
Key Insights and Ideas
- The "Variance Trap": Stochasticity alone can cause performance swings of up to 1.5 points in accuracy. Without multiple trials, practitioners risk "tuning" to noise rather than actual architectural improvements.
- Filter Size is King: The most critical parameter to tune is the filter region size. While the "optimal" size varies by dataset (longer sentences often benefit from larger filters), a search range of 1–10 is a solid starting point.
- 1-Max Pooling is Sufficient: Despite more complex strategies like $k$-max pooling, simple 1-max pooling consistently performed best.
- Diminishing Returns on Feature Maps: Increasing the number of feature maps helps up to a point (typically 100–600). Beyond that, the risk of overfitting increases with negligible gains.
- Word Vectors Matter: Using pre-trained vectors (word2vec or GloVe) is essential. "Non-static" versions, where vectors are fine-tuned during training, almost always outperform "static" ones.
Key Results
- Activation Functions: ReLU and Tanh are the most reliable choices. Interestingly, using no activation function (Identity) sometimes performed surprisingly well, suggesting linear transformations can capture significant signal in one-layer models.
- Regularization: Dropout had a small beneficial effect (optimal rates between 0.1 and 0.5), but $L_2$ norm constraints had almost no impact on performance.
- Practical Advice: The authors recommend a "coarse line-search" over a single filter size first to find the sweet spot, then exploring combinations of sizes near that value.
Limitations and Future Work
The study is strictly limited to one-layer CNNs. While these are strong baselines, the insights might not perfectly translate to deeper, multi-layer architectures. Additionally, the analysis focuses on sentence-level classification; for longer documents, different architectures might be more appropriate. Future directions include using these empirical ranges to inform automated search methods, such as Bayesian optimization, by narrowing the search space to "reasonable" values.
Summary updated