Variational Pretraining for Semi-supervised Text Classification

Suchin Gururangan, Tam Dang, Dallas Card, Noah A. Smith

Abstract

We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.

Introduction

Modern Natural Language Processing (NLP) is dominated by large-scale pretrained models like BERT and ELMo. While these models achieve state-of-the-art results, they require significant computational resources, massive general-purpose datasets, and often struggle with niche domains unless extensively fine-tuned. In resource-constrained environments—where labeled data is scarce, computing power is limited, and the text domain is specialized—these "high-resource" methods are often impractical. We propose VAMPIRE (VAriational Methods for Pretraining In Resource-limited Environments) to bridge this gap.

Methodology: The VAMPIRE Framework

VAMPIRE adopts a lightweight approach by treating documents as a "bag-of-words" (unigram counts) rather than modeling complex sequences. The core of the framework is a Variational Autoencoder (VAE) pretrained on in-domain, unlabeled data.

The VAMPIRE architecture, illustrating VAE pretraining on unlabeled text and downstream classification using latent states.

The architecture consists of an encoder that maps word frequencies to a latent "topic" space (a Gaussian distribution) and a decoder that attempts to reconstruct the original frequencies. Once pretrained, the VAE's internal states are frozen and used as features for a simple Deep Averaging Network classifier.

A critical component of VAMPIRE is its model selection strategy. Unlike traditional VAEs that optimize for reconstruction loss (Negative Log-Likelihood, or NLL), VAMPIRE utilizes Topic Coherence, measured by Normalized Pointwise Mutual Information (NPMI). Research indicates that models with higher NPMI provide more semantically meaningful features for downstream classification, even if their reconstruction loss is higher.

Experimental Evaluation

We evaluated VAMPIRE across four diverse datasets: AG News (topic classification), IMDB (sentiment analysis), Yahoo! Answers (15-way topic categorization), and Hatespeech (tweet classification). We simulated low-resource settings by varying the number of labeled examples from 200 to 10,000, while utilizing 75k–125k unlabeled in-domain documents for pretraining.

Results and Discussion

VAMPIRE demonstrates significant performance gains in ultra-low-resource settings. As shown in Figure 2, with only 200 labeled instances, VAMPIRE consistently outperforms traditional baselines such as self-training and GloVe embeddings.

Accuracy of VAMPIRE vs. baselines across four datasets as the number of labeled instances increases.

Beyond accuracy, VAMPIRE offers substantial efficiency benefits. Pretraining on the IMDB dataset takes approximately 7 minutes on a standard GPU (or 22 minutes on a CPU), compared to over 12 hours for ELMo on a high-end GPU. Furthermore, VAMPIRE's memory footprint is roughly 750MB, a fraction of the 8.5GB required by ELMo.

While VAMPIRE's bag-of-words approach ignores word order—making it less suitable for tasks requiring syntactic analysis—it provides a highly effective "best-effort" solution for broad topic and sentiment classification in resource-limited scenarios. If resources allow for full fine-tuning of models like BERT, they may still provide superior results, but VAMPIRE serves as a powerful alternative for constrained environments.

Conclusion

VAMPIRE proves that in-domain pretraining on a small scale can rival massive general-purpose models when labels are scarce. By prioritizing topic coherence and computational efficiency, VAMPIRE enables high-performance text classification on standard hardware with minimal labeled data.

Summary updated

Abstract

We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.
BibTeX
@inproceedings{gururangan2019variational,
  author = {Suchin Gururangan and Tam Dang and Dallas Card and Noah A. Smith},
  title = {{Variational Pretraining for Semi-supervised Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=LmKvy4ai-Bo},
}
Figure