Abstract
Topic-VQ-VAE (TVQ-VAE) integrates traditional topic modeling with modern Vector-Quantized Variational Auto-Encoders (VQ-VAE) by treating discrete codebooks as "conceptual words." This framework enables flexible, topic-guided generation for both Bag-of-Words text and high-dimensional data like images, bridging the gap between interpretable thematic discovery and powerful generative architectures.
Introduction
Traditional topic modeling, exemplified by Latent Dirichlet Allocation (LDA), remains a cornerstone for discovering thematic structures in text. However, these models are primarily restricted to the "Bag-of-Words" (BoW) paradigm, which fails to capture the rich semantic information available in modern pre-trained language models or generate complex, non-textual data. Conversely, modern generative models utilize Vector-Quantized (VQ) embeddings to discretize continuous data but often lack the explicit, interpretable thematic structure inherent in topic models.
TVQ-VAE addresses this gap by extracting topics compatible with high-dimensional embeddings, providing a flexible framework to guide diverse types of document generation.
Methodology
The core innovation of TVQ-VAE is the reinterpretation of the discrete latent codebook of a VQ-VAE as a set of "conceptual words."
The model operates through three primary components:
- Conceptual Bag-of-Words: VQ codes are treated as a vocabulary of abstract concepts rather than mere compressed data points.
- Generative Topic Framework: A topic-modeling layer is integrated into the VQ-VAE architecture. A latent variable $\theta$ represents the document's topic distribution, influencing the selection of conceptual words.
- Flexible Decoding: Built on a VQ-VAE backbone, the model can generate traditional BoW distributions for document analysis or use autoregressive priors (like PixelCNN or Transformers) to generate full images based on specific topics.
Experimental Setup
The authors evaluated TVQ-VAE across two distinct domains:
- Text Analysis: Benchmarked using the 20 Newsgroups (20NG) dataset (~16k documents, 20 labels) and the New York Times (NYT) dataset (~32k documents, 28.7k vocabulary).
- Image Generation: Evaluated using CelebA (~200k facial images) to cluster features like hair color, and CIFAR-10 (60k images) for diverse object class generation.
Results and Discussion
TVQ-VAE demonstrates that VQ-VAE codebooks are naturally suited for topic learning. By quantizing the embedding space, the model creates a "vocabulary" that the topic layer organizes into themes. Notably, in NYT experiments, the model achieved competitive results using only 300 "virtual" codebooks—less than 1% of the original vocabulary size.
Key quantitative findings include:
- Topic Quality (TQ): On the NYT dataset, TVQ-VAE outperformed traditional BoW models (LDA, ProdLDA) and embedding-based models (ETM) in both coherence and diversity.
- Document Representation: TVQ-VAE showed robust performance in clustering tasks (NMI and Purity), scaling effectively with vocabulary size.
- Image Generation: On CIFAR-10, the model achieved lower Negative Log-Likelihood (NLL) than unsupervised baselines, suggesting that explicit topic extraction aids the generative prior in understanding underlying data structures.
The model also allows for "Visualizing Topics through Synthesis." Instead of reading word lists, users can generate grids of images sharing topical attributes. Furthermore, "Reference-Based Generation" enables the model to extract a topic distribution from a reference image and generate new images sharing the same semantic theme.
Conclusion
TVQ-VAE successfully bridges the gap between discrete conceptual discovery and high-fidelity generation. While performance on very small datasets like 20NG reaches a saturation point, the model's ability to scale and its flexibility across modalities offer a promising path forward. Future work will explore multi-modal generation, using topics as the semantic glue between disparate data types.
Summary updated