Variational Pretraining for Semi-supervised Text Classification

Suchin Gururangan, Tam Dang, Dallas Card, Noah A. Smith

Authors: Suchin Gururangan, Tam Dang, Dallas Card, Noah A. Smith. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

Introduction

In the current NLP landscape, pretraining typically involves massive Transformer-based models like BERT or ELMo, which require web-scale datasets and significant computational resources. This "high-resource bias" creates a barrier for researchers working in niche domains or social sciences who may only have access to a few hundred labeled examples and limited hardware.

This paper introduces VAMPIRE (Variational Methods for Pretraining In Resource-limited Environments), a lightweight framework designed to bring the benefits of semi-supervised pretraining to low-resource settings. VAMPIRE achieves competitive performance using only a CPU and a modest amount of in-domain, unlabeled data.

The VAMPIRE Framework

VAMPIRE adapts variational document models for pretraining by focusing on document-level semantics rather than sequence-level word order.

Overview of the VAMPIRE architecture, from VAE pretraining on unlabeled text to downstream classification.

Architecture and Training

  1. Variational Autoencoder (VAE): The core of VAMPIRE is a bag-of-words VAE. It takes word frequencies as input, compresses them into a latent space via a multi-layer perceptron (MLP) encoder, and attempts to reconstruct the original frequencies through a decoder.
  2. Feature Extraction: Once pretrained on unlabeled in-domain text, the internal states of the VAE encoder—specifically the hidden layers and the latent representation—are extracted as features.
  3. Downstream Classification: These VAMPIRE embeddings are concatenated with standard word vectors and fed into a simple Deep Averaging Network (DAN) classifier. This allows the model to leverage both general word-level information and domain-specific document representations.

Model Selection via Topic Coherence

A critical finding of the authors is that standard VAE evaluation metrics, such as Negative Log Likelihood (NLL), do not correlate well with downstream classification accuracy. Instead, the authors utilize Topic Coherence (NPMI)—a measure of semantic relatedness among the top words in a latent "topic"—as an early stopping criterion. This ensures the model is selected when its internal features are most semantically meaningful.

Experimental Results

The model was evaluated across four diverse datasets: AG News (topic), Hatespeech (tweets), IMDB (sentiment), and Yahoo! Answers (topic). To simulate low-resource environments, the authors varied the number of labeled instances from 200 to 10,000.

Classification accuracy of VAMPIRE compared to self-training and GloVe baselines across four datasets as labeled data increases.

In the most constrained settings (200 labeled examples), VAMPIRE consistently outperformed traditional baselines like self-training and GLoVe. For instance, on the AG News dataset with only 200 labels, VAMPIRE achieved 83.9% accuracy, significantly higher than the 77.3% achieved by self-training. Furthermore, VAMPIRE often rivals or exceeds the performance of much larger models (like frozen ELMo) when those models are trained on generic rather than in-domain data.

Efficiency and Accessibility

VAMPIRE is designed for accessibility. While ELMo contains approximately 159 million parameters, VAMPIRE contains only 3.8 million. This efficiency translates to significant speed gains: pretraining on the IMDB dataset takes roughly 22 minutes on a CPU, compared to over 12 hours on a GPU for ELMo. With a memory footprint of approximately 750MB, VAMPIRE can be deployed on consumer-grade hardware.

Conclusion

VAMPIRE demonstrates that for text classification, a well-trained bag-of-words variational model can be a powerful and efficient alternative to massive sequence models. By prioritizing in-domain pretraining and semantically-driven model selection, VAMPIRE provides a viable path for high-quality NLP in resource-limited environments. Future work may explore how to mitigate the loss of syntactic context inherent in bag-of-words approaches and improve handling of out-of-vocabulary terms.

Summary updated

BibTeX
@inproceedings{gururangan2019variational,
  author = {Suchin Gururangan and Tam Dang and Dallas Card and Noah A. Smith},
  title = {{Variational Pretraining for Semi-supervised Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=LmKvy4ai-Bo},
}
Figure