Abstract
We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" step for a later supervised sequence learning algorithm. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.
Introduction
In 2015, Recurrent Neural Networks (RNNs) were recognized for their theoretical potential in sequential modeling but were notoriously difficult to train in practice. Issues such as vanishing and exploding gradients made them highly sensitive to hyperparameters and optimization challenges, particularly for long sequences. Consequently, simpler models like Bag-of-Words, Support Vector Machines (SVMs), or Convolutional Neural Networks (CNNs) were often preferred for text classification tasks. The central problem addressed by Dai and Le is how to stabilize Long Short-Term Memory (LSTM) networks to learn meaningful representations from long documents, even when labeled data is scarce.
Methods
The authors introduce a "pretrain-then-finetune" paradigm for sequences, proposing two unsupervised methods to initialize LSTM weights before supervised training:
- Recurrent Language Model (LM-LSTM): The network is trained to predict the next token in a sequence, forcing it to learn linguistic syntax and semantics.
- Sequence Autoencoder (SA-LSTM): An encoder LSTM reads the input sequence into a fixed-length hidden vector, and a decoder LSTM attempts to reconstruct the original sequence from this vector.
Following pretraining, the LSTM weights serve as the starting point for a classifier, which is then fine-tuned on labeled data. The researchers also explored Linear Label Gain, a technique that gradually introduces supervised signals at every timestep to improve gradient flow during the early stages of training.
Key Insights and Results
The primary finding was that unsupervised pretraining acts as a powerful regularizer. By starting in a sensible region of the weight space, pretrained LSTMs become robust to variations in sequence length and hidden layer size. Furthermore, the study demonstrated the utility of "outside" data; for instance, pretraining on millions of unlabeled Amazon reviews significantly boosted performance on the smaller Rotten Tomatoes dataset.
The semi-supervised LSTMs consistently outperformed randomly initialized models and established new state-of-the-art results across several benchmarks.
As shown in the results, the SA-LSTM achieved a 7.24% error rate on IMDB, surpassing Paragraph Vectors. On DBpedia, it reached a 1.19% error rate, outperforming deep CNNs. The sequence autoencoder generally proved more effective for long-term dependencies than the language model, likely because the reconstruction task requires preserving information about the entire sequence structure.
Conclusion and Future Work
While pretraining on massive unlabeled corpora involves a significant initial computational cost, it is a one-time investment that yields substantial gains in stability and accuracy. The authors noted that joint training (simultaneous supervised and unsupervised learning) was less effective than sequential pretraining, possibly due to competing objectives during optimization. This research was a critical precursor to the modern Transformer-based era of NLP, establishing the fundamental idea that large-scale unsupervised pretraining is the most effective foundation for high-performance sequence models.
Summary updated