Convolutional Neural Networks for Sentence Classification

Yoon Kim

Abstract We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.

Introduction

In 2014, the field of Natural Language Processing (NLP) was transitioning toward deep learning. However, many high-performing models at the time relied on complex architectures, such as Recursive Neural Networks requiring expensive parse trees, or extensive feature engineering. This paper, authored by Yoon Kim, explores whether the success of Convolutional Neural Networks (CNNs) in computer vision—specifically their ability to automatically learn spatial hierarchies of features—could be replicated in NLP. The central hypothesis was that pre-trained word embeddings, such as Google’s word2vec, could serve as a "universal" foundation for diverse tasks like sentiment analysis and question classification, even with minimal task-specific tuning.

Model Architecture

The proposed model architecture is characterized by its simplicity and efficiency, consisting of four primary stages:

  1. Input Layer: A sentence is represented as a matrix of size $n \times k$, where $n$ is the sentence length and each row is a $k$-dimensional word vector.
  2. Convolutional Layer: The model employs 1D filters that slide over full rows (words) rather than the 2D filters typical in image processing. Multiple filter widths (e.g., 3, 4, or 5 words) are used to capture various n-gram features.
  3. Max-over-time Pooling: A max-pooling operation is applied over the output of each filter. This captures the most salient feature for each filter across the entire sentence and enables the model to process sentences of varying lengths.
  4. Fully Connected Layer: The pooled features are passed through a fully connected layer with a softmax activation for final classification.
Model architecture for sentence classification.

The Multichannel Approach

A key innovation introduced is the "multichannel" architecture, inspired by RGB channels in computer vision. The model utilizes two sets of word vectors: a "static" set that remains frozen during training and a "non-static" set that is fine-tuned via backpropagation. This dual-channel approach allows the model to maintain general semantic relationships while simultaneously adapting to the specific nuances of the target task.

Experimental Setup

The model was evaluated across seven diverse benchmarks, ranging from sentiment analysis to question classification:

  • MR: Movie reviews (binary sentiment).
  • SST-1 & SST-2: Stanford Sentiment Treebank (fine-grained and binary).
  • Subj: Subjectivity dataset (subjective vs. objective).
  • TREC: Question classification (6 question types).
  • CR: Customer reviews of various products.
  • MPQA: Opinion polarity detection.

Results and Discussion

The simple CNN architecture achieved state-of-the-art (SOTA) results on 4 out of the 7 tasks at the time of publication. Notably, the CNN-non-static model achieved 93.6% accuracy on TREC, while the CNN-multichannel model reached 88.1% on SST-2.

Performance comparison of CNN variants against baseline models.

Key Insights

  • Efficacy of Pre-training: Even the "static" CNN variant, where word vectors were not updated, performed remarkably well. This confirms that pre-trained embeddings are powerful universal feature extractors.
  • Impact of Fine-tuning: Comparing static and non-static vectors revealed that fine-tuning allows the model to learn task-specific semantics. For instance, sentiment-based fine-tuning can adjust the proximity of words like "good" and "nice" to better reflect degrees of positivity.
  • Structural Simplicity: The CNN outperformed more complex recursive models, suggesting that for sentence-level tasks, local n-gram patterns captured by convolutions are often more critical than global tree structures.
  • Regularization: Dropout was identified as a crucial regularizer, consistently improving performance by 2–4% by preventing over-reliance on specific features.

Conclusion and Future Work

While the CNN models demonstrated significant gains, the multichannel approach yielded mixed results, not always outperforming the single-channel non-static model. This suggests a need for improved regularization techniques during fine-tuning. Additionally, the fixed-width filters may struggle with long-distance dependencies compared to architectures like Transformers. Future research could explore more sophisticated initialization methods for words absent from pre-trained sets to further enhance performance.

Summary updated

Abstract

We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.
BibTeX
@inproceedings{kim2014convolutional,
  author = {Yoon Kim},
  title = {{Convolutional Neural Networks for Sentence Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  url = {https://openreview.net/forum?id=S1-1UGM_WB},
}
Figure