Abstract
This paper introduces a novel approach for topic modeling utilizing latent codebooks from Vector-Quantized Variational Auto-Encoder (VQ-VAE), discretely encapsulating the rich information of pre-trained embeddings such as those from language models. By interpreting latent codebooks and embeddings as a "conceptual bag-of-words," we propose Topic-VQ-VAE (TVQ-VAE), a generative topic model that inversely generates original documents related to respective latent codebooks. TVQ-VAE can visualize topics using various generative distributions, including traditional Bag-of-Words (BoW) and autoregressive image generation. Experimental results in document analysis and image generation demonstrate that TVQ-VAE effectively captures topic context, revealing underlying dataset structures and supporting flexible document generation.
Introduction
Traditional topic modeling, exemplified by Latent Dirichlet Allocation (LDA), remains a cornerstone for discovering thematic structures in text. However, these models often struggle to integrate the high-dimensional richness of modern pre-trained embeddings (e.g., BERT) and are typically restricted to generating word-count histograms rather than structured data. Conversely, modern generative models like VQ-VAEs excel at high-fidelity synthesis but lack explicit, interpretable topical control.
Topic-VQ-VAE (TVQ-VAE) bridges this gap by reimagining the discrete latent codebooks of VQ-VAEs as a "conceptual bag-of-words." By treating these codebooks as a virtual vocabulary, TVQ-VAE creates a flexible generative framework capable of performing traditional topic modeling while simultaneously enabling topic-guided, high-resolution image generation.
Methodology
The core innovation of TVQ-VAE lies in its interpretation of the VQ-VAE framework through a topic-modeling lens.
Conceptual Bag-of-Words (BoW)
In a standard VQ-VAE, an input is mapped to a sequence of discrete codes from a codebook. TVQ-VAE treats these codes as "conceptual words." To overcome the limitation of small codebook sizes compared to natural language vocabularies, the authors employ a "multi-hot" relaxation. Instead of a one-to-one mapping, $K$-nearest embeddings represent a word, effectively expanding the virtual vocabulary.
The Generative Process
The model defines a joint distribution over topics, latent codes, and the final output. For text, it functions as a neural topic model, clustering documents and extracting coherent word sets. For images, it utilizes an autoregressive prior (such as a Transformer) to generate a sequence of latent codes conditioned on a discovered topic vector ($\theta$).
Inference and Semantic Anchoring
The model uses Variational Bayes to decouple the topic distribution from specific latent codes, allowing it to learn the dataset's underlying thematic structure. A key insight is that the discrete latent space of a VQ-VAE acts as a semantic map. By treating these points as "words," the mathematical machinery of topic modeling can be applied to non-textual data. This allows for a variable number of topics not strictly capped by the codebook size.
Experiments and Results
The authors validated TVQ-VAE across text and image domains using datasets including 20 Newsgroups (20NG), New York Times (NYT), CIFAR-10, CelebA, and FacesHQ.
Text Analysis and Scalability
On the NYT dataset, TVQ-VAE demonstrated superior scalability, achieving better Topic Quality (TQ) and document representation (measured by Km-NMI) than established baselines like BerTopic, particularly as vocabulary size increased.
Image Generation and Coherence
For CIFAR-10, the model achieved lower Negative Log-Likelihood (NLL) than unsupervised VQ-VAE baselines, suggesting that explicit topic modeling helps the generator grasp global data structures better than local pixel correlations alone. Qualitatively, "visual topics" successfully clustered images by semantic features—for instance, grouping outdoor landscapes with blue skies separately from close-up portraits.
Topic-Guided Generation
Because the model extracts a topic distribution ($\theta$) from an input, it enables "reference" generation. One can feed a reference image, extract its topic, and generate new images sharing the same theme (e.g., lighting, color palettes) without direct copying.
Conclusion and Future Work
TVQ-VAE offers a powerful framework for interpretable generative modeling. While current experiments often utilized fixed encoders for efficiency, future work may explore end-to-end training for tighter integration. Additionally, the authors suggest extending this approach to multi-modal generation, such as a "topic-to-image" pipeline where text-based topics guide visual synthesis.
Summary updated