A Comparison of Neural Network Training Methods for Text Classification

Anderson de AndradearXiv 2019 · OpenReview

Abstract

We study the impact of neural networks in text classification. Our focus is on training deep neural networks with proper weight initialization and greedy layer-wise pretraining. Results are compared with 1-layer neural networks and Support Vector Machines. We work with a dataset of labeled messages from the Twitter microblogging service and aim to predict weather conditions. A feature extraction procedure specific for the task is proposed, which applies dimensionality reduction using Latent Semantic Analysis. Our results show that neural networks outperform Support Vector Machines with Gaussian kernels, noticing performance gains from introducing additional hidden layers with nonlinearities. The impact of using Nesterov's Accelerated Gradient in backpropagation is also studied. We conclude that deep neural networks are a reasonable approach for text classification and propose further ideas to improve performance.

Introduction

While deep neural networks (DNNs) have revolutionized fields like computer vision, their application to short-form, noisy text—such as Twitter data—remains challenging. The primary difficulty lies in the sparsity and high dimensionality of text data, which often leads to vanishing gradients or saturated units during training. This paper investigates whether modern deep learning techniques, specifically normalized weight initialization and unsupervised pretraining, can enable deep networks to capture complex semantic features and outperform traditional methods like Support Vector Machines (SVMs).

Methodology

Feature Extraction and Dimensionality Reduction

Tweets are notoriously noisy and contain non-standard English. To address this, a custom NLP pipeline was developed to normalize text before feeding it into the models. This process includes hashtag segmentation, specialized Part-of-Speech (POS) tagging, and lemmatization using WordNet. To handle the high dimensionality of the resulting bag-of-words representation, the pipeline applies TF-IDF weighting followed by Singular Value Decomposition (SVD). This reduces the feature space from over 8,000 words to a dense 1,000-dimensional vector, effectively performing Latent Semantic Analysis.

The NLP feature extraction and dimensionality reduction pipeline.

Model Architectures and Optimization

The study compares several architectures:

  1. 1-Layer Neural Network: A shallow baseline using sigmoid activations and random Gaussian initialization.
  2. 3-Layer DNN (Normalized Initialization): A deeper network using tanh activations and the initialization scheme proposed by Glorot & Bengio to maintain variance across layers.
  3. 3-Layer DNN (Greedy Pretraining): A network using Deep Belief Network (DBN) pretraining to initialize weights in a region favorable for gradient-based optimization.
  4. SVM Baseline: An SVM with a non-linear Gaussian (RBF) kernel.

Optimization was performed using backpropagation with a focus on momentum. The study specifically evaluated the impact of Classical Momentum (CM) and Nesterov’s Accelerated Gradient (NAG) on convergence speed and final error.

Key Results

The experimental results demonstrate that neural networks are highly effective for text classification, even on microblogging data.

Deep vs. Shallow Performance

The deep architectures consistently outperformed the shallow 1-layer baseline. As shown in the training curves, both normalized initialization and greedy layer-wise pretraining allowed the 3-layer networks to reach lower error rates than the 1-layer model.

Training and validation RMSE for various neural network architectures over 30 epochs.

The 3-layer network with DBN pretraining emerged as the top performer, achieving a validation RMSE of 0.1328. This model also performed competitively in a real-world benchmark, placing in the top 10% of a Kaggle competition with a test RMSE of 0.15638.

Comparison with SVMs

Neural networks significantly outperformed the SVM baseline. The 1-layer NN achieved an average RMSE of 0.1340, while the SVM lagged behind at 0.1567. This suggests that the dense representations learned by neural networks are superior to the kernel-based approach for this specific high-dimensional task.

Optimization Insights

The use of momentum was found to be critical. Both NAG and CM allowed the networks to converge to lower error rates in fewer iterations compared to standard gradient descent.

Conclusion and Future Work

This research confirms that deep neural networks, when properly initialized or pretrained, are a robust choice for text classification on noisy datasets. While unsupervised pretraining provides a significant boost, its advantages diminish as labeled data becomes more abundant and overfitting begins.

Future work should explore advanced regularization techniques such as dropout or denoising autoencoders to mitigate overfitting. Additionally, moving beyond the bag-of-words model to include bigrams could better preserve the contextual meaning necessary for nuanced tasks like sentiment analysis.

Summary updated

Abstract

We study the impact of neural networks in text classification. Our focus is on training deep neural networks with proper weight initialization and greedy layer-wise pretraining. Results are compared with 1-layer neural networks and Support Vector Machines. We work with a dataset of labeled messages from the Twitter microblogging service and aim to predict weather conditions. A feature extraction procedure specific for the task is proposed, which applies dimensionality reduction using Latent Semantic Analysis. Our results show that neural networks outperform Support Vector Machines with Gaussian kernels, noticing performance gains from introducing additional hidden layers with nonlinearities. The impact of using Nesterov's Accelerated Gradient in backpropagation is also studied. We conclude that deep neural networks are a reasonable approach for text classification and propose further ideas to improve performance.
BibTeX
@misc{andrade2019comparison,
  author = {Anderson de Andrade},
  title = {{A Comparison of Neural Network Training Methods for Text Classification}},
  year = {2019},
  eprint = {1910.12674},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.12674},
}
Figure