Semi-supervised Variational Autoencoders for Sequence Classification

Weidi Xu, Haoze SunarXiv 2016 · arXiv

Introduction

Semi-supervised learning represents a significant opportunity for text classification, where labeled data is often scarce and expensive, but unlabeled text is plentiful. While Variational Autoencoders (VAEs) have proven highly effective for semi-supervised image classification, their application to sequential data like text has historically been problematic.

The primary obstacle is a phenomenon known as label ignoring. In a standard VAE framework, the decoder's objective is to reconstruct the input sequence. However, powerful sequence models like LSTMs are so effective at local language modeling—predicting the next word based on preceding context—that they often learn to ignore the global categorical label provided during training. When the decoder ignores this label, it fails to provide the necessary "teaching signal" required to improve the classifier using unlabeled data.

Methodology: The SSVAE Framework

To address the label-ignoring problem, the Semi-supervised Sequential Variational Autoencoder (SSVAE) introduces structural modifications to ensure the decoder remains conditioned on the category label throughout the generation process.

The SSVAE architecture featuring a classifier, encoder, and a decoder that receives label information at every time step.

The authors propose two specific Conditional LSTM (CLSTM) architectures:

  • CLSTM-I: The categorical label vector is concatenated with the word embedding at every time step.
  • CLSTM-II: Label information is injected directly into the LSTM's internal memory cell update equations, providing a stronger inductive bias.

Optimization via Reinforcement Learning

Training on unlabeled data requires marginalizing over all possible labels, which is computationally expensive. The authors reformulate this as a Reinforcement Learning problem: the classifier acts as a policy selecting a label, and the VAE provides a reconstruction-based reward. Gradients are estimated using the REINFORCE algorithm, stabilized by variance reduction techniques (baselines S1 and S2).

Experimental Results

The SSVAE was evaluated on the IMDB sentiment analysis dataset and the AG’s News corpus. A key focus was performance under limited-label regimes.

Error rates on the IMDB dataset across different labeled data sizes and model variants.

As shown in the results, the SSVAE-II model demonstrates remarkable data efficiency. With only 2.5K labeled samples, it achieves a 10.28% error rate, outperforming a standard supervised LSTM trained on 20K samples (10.90%). When augmented with language model pre-training (LM-LSTM), the model achieves state-of-the-art performance, reaching an error rate of 7.23% on the full IMDB benchmark.

Key Insights and Discussion

The success of the SSVAE hinges on the "Discrimination Index," a metric developed by the authors to quantify how effectively a decoder distinguishes between labels. Their analysis confirms that while vanilla LSTMs have low discrimination, the CLSTM structures maintain a high index, forcing the model to utilize label information for reconstruction.

Limitations and Future Work

Despite its strengths, the SSVAE is not a standalone solution. The highest performance tiers still require initialization with pre-trained weights (like LM-LSTM). Furthermore, analysis of the latent space suggests that only a fraction of the latent units (approximately 10 out of 50) are actively utilized, indicating that future work could better optimize how the model captures complex, non-categorical global features.

Conclusion

The SSVAE successfully bridges the gap between VAEs and sequence classification. By enforcing label dependency at every time step and leveraging RL-based optimization, it provides a robust framework for utilizing unlabeled text data, significantly reducing the dependency on large-scale human annotation.

Summary updated

Abstract

Although semi-supervised variational autoencoder (SemiVAE) works in image classification task, it fails in text classification task if using vanilla LSTM as its decoder. From a perspective of reinforcement learning, it is verified that the decoder's capability to distinguish between different categorical labels is essential. Therefore, Semi-supervised Sequential Variational Autoencoder (SSVAE) is proposed, which increases the capability by feeding label into its decoder RNN at each time-step. Two specific decoder structures are investigated and both of them are verified to be effective. Besides, in order to reduce the computational complexity in training, a novel optimization method is proposed, which estimates the gradient of the unlabeled objective function by sampling, along with two variance reduction techniques. Experimental results on Large Movie Review Dataset (IMDB) and AG's News corpus show that the proposed approach significantly improves the classification accuracy compared with pure-supervised classifiers, and achieves competitive performance against previous advanced methods. State-of-the-art results can be obtained by integrating other pretraining-based methods.
BibTeX
@misc{xu2016semi,
  author = {Weidi Xu and Haoze Sun},
  title = {{Semi-supervised Variational Autoencoders for Sequence Classification}},
  year = {2016},
  eprint = {1603.02514},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1603.02514},
}
Figure