Semi-supervised Sequence Learning

Andrew M. Dai, Quoc V. Le

TL;DR

This paper demonstrates that pretraining Long Short-Term Memory (LSTM) networks on unlabeled data—either as language models or sequence autoencoders—significantly improves their stability and performance on supervised tasks. By using this semi-supervised approach, the authors achieved state-of-the-art results in text classification and competitive performance in image recognition.

What Problem Does This Paper Address?

Recurrent Neural Networks (RNNs), specifically LSTMs, are theoretically ideal for sequential data because they can preserve word ordering and context. However, in practice, they are notoriously difficult to train. They suffer from unstable gradients (vanishing or exploding) and are highly sensitive to hyperparameter choices.

Because of these optimization hurdles, simpler models like "bag-of-words" or Paragraph Vectors often outperformed LSTMs on document classification tasks. The authors set out to bridge this gap by asking: Can we use the vast amounts of available unlabeled text to "warm up" an LSTM so that it's easier to train and generalizes better on specific labeled tasks?

Methods

The authors propose two primary "pretraining" algorithms where the weights of an LSTM are initialized using unsupervised tasks before being fine-tuned for a supervised task (like sentiment analysis).

  1. Recurrent Language Model (LM-LSTM): The network is trained to predict the next token in a sequence. This is a classic NLP task that forces the model to learn the structure and semantics of language.
  2. Sequence Autoencoder (SA-LSTM): This is the paper's more novel contribution. An "encoder" LSTM reads an entire input sequence into a hidden state vector. A "decoder" LSTM (sharing the same weights) then attempts to reconstruct the original input sequence from that vector.
Sequence Autoencoder Architecture
  1. Fine-tuning: Once pretrained, the LSTM weights are used to initialize a classifier. The model is then trained on labeled data, allowing the weights to be "fine-tuned" for the specific task.

The authors also experimented with Joint Training (doing both tasks at once) and Linear Label Gain (gradually introducing the supervised signal across the sequence), though pretraining followed by fine-tuning generally yielded the best results.

Datasets

The researchers evaluated their methods across several diverse benchmarks:

  • Sentiment Analysis: IMDB movie reviews and Rotten Tomatoes (short snippets).
  • Text Classification: 20 Newsgroups (emails) and DBpedia (Wikipedia abstracts).
  • Image Recognition: CIFAR-10. To treat images as sequences, they fed the LSTM the image row-by-row (pixels).
  • External Unlabeled Data: They used a massive dataset of 7.9 million Amazon reviews to pretrain models for the Rotten Tomatoes task, demonstrating the power of "transferring" unsupervised knowledge from one domain to another.

Key Insights and Ideas

  • Stability is Key: The most significant "aha moment" wasn't just higher accuracy, but optimization stability. Randomly initialized LSTMs often "break" (gradients explode) if you increase the hidden layer size or the number of backpropagation steps. Pretrained LSTMs (SA-LSTMs) remained stable even as the models grew larger.
  • Autoencoders vs. Language Models: While both help, the sequence autoencoder (SA-LSTM) often outperformed the language model. The authors hypothesize this is because language modeling is a "short-term" objective (predicting the very next word), whereas autoencoding forces the hidden state to capture the "long-term" essence of the entire document.
  • The Power of Related Unlabeled Data: One of the most striking findings was that pretraining on Amazon reviews improved accuracy on Rotten Tomatoes from 79.0% to 83.3%. This suggests that unsupervised learning on a large, related dataset can be as effective as adding a substantial amount of expensive labeled data.

Key Results

The SA-LSTM reached or surpassed previous state-of-the-art results on several text benchmarks:

SA-LSTM Results Summary
  • IMDB: Achieved a 7.24% error rate, outperforming Paragraph Vectors (7.42%).
  • DBpedia: Achieved a 1.19% error rate, beating strong convolutional neural network baselines.
  • Rotten Tomatoes: Reduced the error rate to 16.7% when using Amazon reviews for pretraining.
  • CIFAR-10: While LSTMs are not the standard for images, the pretrained 2-layer LM-LSTM achieved an 18.7% error rate, which was better than some baseline convolutional Deep Belief Networks at the time.

Limitations and Future Work

While the results are impressive, the authors note some limitations:

  • Image Domain: Although the results on CIFAR-10 were good for an RNN, they still lagged behind the best state-of-the-art Convolutional Neural Networks (CNNs), which are architecturally specialized for spatial data.
  • Computational Cost: Pretraining on millions of unlabeled sequences (like the Amazon dataset) requires significant computational resources before the "actual" supervised training even begins.
  • Task Specificity: The paper focuses on classification. It remains to be seen how well these specific pretraining methods translate to other sequence-to-sequence tasks like real-time translation or summarization without further modification.

Summary updated

Abstract

We present two approaches to use unlabeled data to improve Sequence Learningwith recurrent networks. The first approach is to predict what comes next in asequence, which is a language model in NLP. The second approach is to use asequence autoencoder, which reads the input sequence into a vector and predictsthe input sequence again. These two algorithms can be used as a “pretraining”algorithm for a later supervised sequence learning algorithm. In other words, theparameters obtained from the pretraining step can then be used as a starting pointfor other supervised training models. In our experiments, we find that long shortterm memory recurrent networks after pretrained with the two approaches becomemore stable to train and generalize better. With pretraining, we were able toachieve strong performance in many classification tasks, such as text classificationwith IMDB, DBpedia or image recognition in CIFAR-10.
BibTeX
@inproceedings{dai2015semi,
  author = {Andrew M. Dai and Quoc V. Le},
  title = {{Semi-supervised Sequence Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
  url = {https://openreview.net/forum?id=SyZBFwbOWH},
}
Figure