Introduction
Document representation has traditionally been divided between two distinct paradigms: local word embeddings and global topic models. Bag-of-Words (BOW) models suffer from high dimensionality and sparsity, failing to capture semantic relationships between words. Topic models like Latent Dirichlet Allocation (LDA) capture global patterns but ignore local word contexts. Conversely, word embeddings like Word2Vec excel at local semantic relationships but lack a native mechanism for representing entire document themes.
TopicVec (Generative Topic Embedding) bridges this gap by representing topics as vectors in the same continuous space as words. By integrating the local context of neighboring words with the global context of document-wide themes, TopicVec produces dense, low-dimensional document representations that significantly outperform traditional methods.
Methodology
TopicVec is a generative model where topics are represented as embedding vectors ($t_k$) in an $N$-dimensional space shared with word embeddings ($v_w$). The core of the model is an Extended Link Function that predicts a word based on both its local neighbors and a latent topic vector.
The generative process follows a Dirichlet distribution to draw topic assignments, similar to LDA. However, the probability of a word appearing is determined by its similarity to both the local context and the document's assigned topic vector. To infer parameters, the authors utilize a Variational GEM Algorithm (Generalized Expectation-Maximization) using mean-field variational inference. Learning is conducted in stages: word embeddings are pre-learned and treated as constants while topic-specific parameters are optimized.
A key architectural innovation is the "Null Topic," where one topic vector is fixed to zero. This captures background noise and non-thematic words, enhancing the purity of the learned topical embeddings.
Experimental Setup
The model was evaluated on two standard benchmarks:
- 20 Newsgroups: Approximately 20,000 documents across 20 categories.
- Reuters-21578 (ApteMod): The 10 largest categories, totaling 8,025 documents.
Key Results and Discussion
TopicVec demonstrates significant improvements in both classification accuracy and representation efficiency.
Superior Classification and Compactness
As shown in the results table, TopicVec and its variant (TV+MeanWV) consistently outperform established methods like LDA, sLDA, Doc2Vec, and Topical Word Embedding (TWE). Notably, TopicVec achieves these results with extreme compactness. For the 20News dataset, TopicVec requires only 281 features, a massive reduction from the 50,381 features used by BOW models.
Semantic Richness and Speed
By leveraging pre-trained word embeddings, TopicVec can extract coherent topics even from a single document—a task where traditional topic models typically fail due to data sparsity. Furthermore, the model is computationally efficient; while embedding-based models like Gaussian LDA can take days to converge, TopicVec completes 100 EM iterations in approximately two hours.
Conclusion
TopicVec successfully unifies local and global semantic contexts into a single generative framework. By treating topics as "semantic centroids" in a continuous embedding space, it provides a powerful and efficient alternative to traditional document modeling. Future work may explore joint representations that combine topic proportions with the embeddings themselves, potentially further enhancing performance in downstream tasks like retrieval and summarization.
Summary updated