Abstract
We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.
Introduction
Modern Natural Language Processing (NLP) is dominated by large-scale pretrained models like BERT and ELMo. While these models achieve state-of-the-art results, they require significant computational resources, massive general-purpose datasets, and often struggle with niche domains unless extensively fine-tuned. In resource-constrained environments—where labeled data is scarce, computing power is limited, and the text domain is specialized—these "high-resource" methods are often impractical. We propose VAMPIRE (VAriational Methods for Pretraining In Resource-limited Environments) to bridge this gap.
Methodology: The VAMPIRE Framework
VAMPIRE adopts a lightweight approach by treating documents as a "bag-of-words" (unigram counts) rather than modeling complex sequences. The core of the framework is a Variational Autoencoder (VAE) pretrained on in-domain, unlabeled data.
The architecture consists of an encoder that maps word frequencies to a latent "topic" space (a Gaussian distribution) and a decoder that attempts to reconstruct the original frequencies. Once pretrained, the VAE's internal states are frozen and used as features for a simple Deep Averaging Network classifier.
A critical component of VAMPIRE is its model selection strategy. Unlike traditional VAEs that optimize for reconstruction loss (Negative Log-Likelihood, or NLL), VAMPIRE utilizes Topic Coherence, measured by Normalized Pointwise Mutual Information (NPMI). Research indicates that models with higher NPMI provide more semantically meaningful features for downstream classification, even if their reconstruction loss is higher.
Experimental Evaluation
We evaluated VAMPIRE across four diverse datasets: AG News (topic classification), IMDB (sentiment analysis), Yahoo! Answers (15-way topic categorization), and Hatespeech (tweet classification). We simulated low-resource settings by varying the number of labeled examples from 200 to 10,000, while utilizing 75k–125k unlabeled in-domain documents for pretraining.
Results and Discussion
VAMPIRE demonstrates significant performance gains in ultra-low-resource settings. As shown in Figure 2, with only 200 labeled instances, VAMPIRE consistently outperforms traditional baselines such as self-training and GloVe embeddings.
Beyond accuracy, VAMPIRE offers substantial efficiency benefits. Pretraining on the IMDB dataset takes approximately 7 minutes on a standard GPU (or 22 minutes on a CPU), compared to over 12 hours for ELMo on a high-end GPU. Furthermore, VAMPIRE's memory footprint is roughly 750MB, a fraction of the 8.5GB required by ELMo.
While VAMPIRE's bag-of-words approach ignores word order—making it less suitable for tasks requiring syntactic analysis—it provides a highly effective "best-effort" solution for broad topic and sentiment classification in resource-limited scenarios. If resources allow for full fine-tuning of models like BERT, they may still provide superior results, but VAMPIRE serves as a powerful alternative for constrained environments.
Conclusion
VAMPIRE proves that in-domain pretraining on a small scale can rival massive general-purpose models when labels are scarce. By prioritizing topic coherence and computational efficiency, VAMPIRE enables high-performance text classification on standard hardware with minimal labeled data.
Summary updated