Abstract
We study the impact of neural networks in text classification. Our focus is on training deep neural networks with proper weight initialization and greedy layer-wise pretraining. Results are compared with 1-layer neural networks and Support Vector Machines. We work with a dataset of labeled messages from the Twitter microblogging service and aim to predict weather conditions. A feature extraction procedure specific for the task is proposed, which applies dimensionality reduction using Latent Semantic Analysis. Our results show that neural networks outperform Support Vector Machines with Gaussian kernels, noticing performance gains from introducing additional hidden layers with nonlinearities. The impact of using Nesterov's Accelerated Gradient in backpropagation is also studied. We conclude that deep neural networks are a reasonable approach for text classification and propose further ideas to improve performance.
Introduction
While deep neural networks (DNNs) have revolutionized fields like computer vision, their application to short-form, noisy text—such as Twitter data—remains challenging. The primary difficulty lies in the sparsity and high dimensionality of text data, which often leads to vanishing gradients or saturated units during training. This paper investigates whether modern deep learning techniques, specifically normalized weight initialization and unsupervised pretraining, can enable deep networks to capture complex semantic features and outperform traditional methods like Support Vector Machines (SVMs).
Methodology
Feature Extraction and Dimensionality Reduction
Tweets are notoriously noisy and contain non-standard English. To address this, a custom NLP pipeline was developed to normalize text before feeding it into the models. This process includes hashtag segmentation, specialized Part-of-Speech (POS) tagging, and lemmatization using WordNet. To handle the high dimensionality of the resulting bag-of-words representation, the pipeline applies TF-IDF weighting followed by Singular Value Decomposition (SVD). This reduces the feature space from over 8,000 words to a dense 1,000-dimensional vector, effectively performing Latent Semantic Analysis.
Model Architectures and Optimization
The study compares several architectures:
- 1-Layer Neural Network: A shallow baseline using sigmoid activations and random Gaussian initialization.
- 3-Layer DNN (Normalized Initialization): A deeper network using
tanhactivations and the initialization scheme proposed by Glorot & Bengio to maintain variance across layers. - 3-Layer DNN (Greedy Pretraining): A network using Deep Belief Network (DBN) pretraining to initialize weights in a region favorable for gradient-based optimization.
- SVM Baseline: An SVM with a non-linear Gaussian (RBF) kernel.
Optimization was performed using backpropagation with a focus on momentum. The study specifically evaluated the impact of Classical Momentum (CM) and Nesterov’s Accelerated Gradient (NAG) on convergence speed and final error.
Key Results
The experimental results demonstrate that neural networks are highly effective for text classification, even on microblogging data.
Deep vs. Shallow Performance
The deep architectures consistently outperformed the shallow 1-layer baseline. As shown in the training curves, both normalized initialization and greedy layer-wise pretraining allowed the 3-layer networks to reach lower error rates than the 1-layer model.
The 3-layer network with DBN pretraining emerged as the top performer, achieving a validation RMSE of 0.1328. This model also performed competitively in a real-world benchmark, placing in the top 10% of a Kaggle competition with a test RMSE of 0.15638.
Comparison with SVMs
Neural networks significantly outperformed the SVM baseline. The 1-layer NN achieved an average RMSE of 0.1340, while the SVM lagged behind at 0.1567. This suggests that the dense representations learned by neural networks are superior to the kernel-based approach for this specific high-dimensional task.
Optimization Insights
The use of momentum was found to be critical. Both NAG and CM allowed the networks to converge to lower error rates in fewer iterations compared to standard gradient descent.
Conclusion and Future Work
This research confirms that deep neural networks, when properly initialized or pretrained, are a robust choice for text classification on noisy datasets. While unsupervised pretraining provides a significant boost, its advantages diminish as labeled data becomes more abundant and overfitting begins.
Future work should explore advanced regularization techniques such as dropout or denoising autoencoders to mitigate overfitting. Additionally, moving beyond the bag-of-words model to include bigrams could better preserve the contextual meaning necessary for nuanced tasks like sentiment analysis.
Summary updated