Generative Topic Embedding: a Continuous Representation of Documents (Extended Version with Proofs)

Shaohua Li, Tat-Seng Chua, Jun Zhu, Chunyan Miao

Introduction

Document representation has traditionally been divided between two distinct paradigms: local word embeddings and global topic models. Bag-of-Words (BOW) models suffer from high dimensionality and sparsity, failing to capture semantic relationships between words. Topic models like Latent Dirichlet Allocation (LDA) capture global patterns but ignore local word contexts. Conversely, word embeddings like Word2Vec excel at local semantic relationships but lack a native mechanism for representing entire document themes.

TopicVec (Generative Topic Embedding) bridges this gap by representing topics as vectors in the same continuous space as words. By integrating the local context of neighboring words with the global context of document-wide themes, TopicVec produces dense, low-dimensional document representations that significantly outperform traditional methods.

Methodology

TopicVec is a generative model where topics are represented as embedding vectors ($t_k$) in an $N$-dimensional space shared with word embeddings ($v_w$). The core of the model is an Extended Link Function that predicts a word based on both its local neighbors and a latent topic vector.

The TopicVec generative graphical model.

The generative process follows a Dirichlet distribution to draw topic assignments, similar to LDA. However, the probability of a word appearing is determined by its similarity to both the local context and the document's assigned topic vector. To infer parameters, the authors utilize a Variational GEM Algorithm (Generalized Expectation-Maximization) using mean-field variational inference. Learning is conducted in stages: word embeddings are pre-learned and treated as constants while topic-specific parameters are optimized.

A key architectural innovation is the "Null Topic," where one topic vector is fixed to zero. This captures background noise and non-thematic words, enhancing the purity of the learned topical embeddings.

Experimental Setup

The model was evaluated on two standard benchmarks:

  1. 20 Newsgroups: Approximately 20,000 documents across 20 categories.
  2. Reuters-21578 (ApteMod): The 10 largest categories, totaling 8,025 documents.

Key Results and Discussion

TopicVec demonstrates significant improvements in both classification accuracy and representation efficiency.

Classification performance comparison on 20Newsgroups and Reuters datasets.

Superior Classification and Compactness

As shown in the results table, TopicVec and its variant (TV+MeanWV) consistently outperform established methods like LDA, sLDA, Doc2Vec, and Topical Word Embedding (TWE). Notably, TopicVec achieves these results with extreme compactness. For the 20News dataset, TopicVec requires only 281 features, a massive reduction from the 50,381 features used by BOW models.

Semantic Richness and Speed

By leveraging pre-trained word embeddings, TopicVec can extract coherent topics even from a single document—a task where traditional topic models typically fail due to data sparsity. Furthermore, the model is computationally efficient; while embedding-based models like Gaussian LDA can take days to converge, TopicVec completes 100 EM iterations in approximately two hours.

Conclusion

TopicVec successfully unifies local and global semantic contexts into a single generative framework. By treating topics as "semantic centroids" in a continuous embedding space, it provides a powerful and efficient alternative to traditional document modeling. Future work may explore joint representations that combine topic proportions with the embeddings themselves, potentially further enhancing performance in downstream tasks like retrieval and summarization.

Summary updated

BibTeX
@inproceedings{li2016generative,
  author = {Shaohua Li and Tat-Seng Chua and Jun Zhu and Chunyan Miao},
  title = {{Generative Topic Embedding: a Continuous Representation of Documents (Extended Version with Proofs)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/p16-1063},
}
Figure