Introduction
Traditional topic models, such as Latent Dirichlet Allocation (LDA), primarily rely on word co-occurrence within documents to infer thematic structures. While effective for long-form text, these models struggle with short, sparse texts like social media posts or headlines. In such contexts, related words (e.g., "physician" and "doctor") may never appear together in the same short document, preventing the model from linking them to a common topic. Furthermore, traditional models are constrained by a fixed vocabulary, making them unable to process out-of-vocabulary (OOV) terms encountered after training.
To address these challenges, we present the Latent Concept Topic Model (LCTM). By leveraging pre-trained word embeddings, LCTM shifts the modeling focus from discrete word counts to continuous vector spaces, enabling more robust topic discovery and the natural handling of OOV words.
Methodology
The core innovation of LCTM is the introduction of "latent concepts" as an intermediate layer between topics and words. Instead of modeling topics as distributions over individual words, LCTM models them as distributions over these latent concepts.
The Generative Process
Each latent concept is represented as a localized Gaussian distribution within a continuous word embedding space (e.g., GloVe). The generative process for a word in a document is as follows:
- Select a topic from the document's topic distribution.
- Select a latent concept from the chosen topic's distribution over concepts.
- Draw a word vector from the selected concept's Gaussian distribution.
This approach allows the model to group semantically similar words—such as "running," "ran," and "went"—into a single concept based on their proximity in the embedding space. Consequently, the model can recognize a topic even if different authors use different synonyms.
Inference and Scalability
For inference, the model utilizes Collapsed Gibbs Sampling. To maintain computational efficiency, the authors implement an optimization where only the $M$ nearest concepts in the embedding space are considered for each word, ensuring the model remains scalable to large vocabularies.
Experimental Setup
The model was evaluated using subsets of the 20Newsgroups corpus, specifically curated to simulate short-text environments:
- Short Text Subsets: Documents with fewer than 50 words, categorized by size (400short, 800short, and 1561short).
- Held-out Set: A set of 7,235 documents used to evaluate performance on OOV words.
- Embeddings: 50-dimensional GloVe vectors pre-trained on Wikipedia were used to define the continuous space.
Results and Discussion
LCTM was compared against several baselines, including standard LDA and the Biterm Topic Model (BTM), which is specifically designed for short texts.
Clustering Performance
In document clustering tasks, measured by Adjusted Mutual Information (AMI), LCTM consistently outperformed all baseline models across all short-text datasets. The "concept" layer proved more effective at capturing thematic structure than word-level co-occurrence.
Handling Out-of-Vocabulary Words
A significant advantage of LCTM is its ability to process unseen words. Because the model operates in a continuous vector space, any word with a pre-trained embedding can be mapped to a latent concept. In classification tests where approximately 35% of the words were OOV, LCTM achieved 30.2% accuracy, a substantial improvement over LDA's 21.5%.
Conclusion
The Latent Concept Topic Model provides a robust framework for topic inference in sparse data environments. By integrating word embeddings and latent concepts, it overcomes the limitations of word co-occurrence and fixed vocabularies. Future work may explore the use of higher-dimensional embeddings and the development of models that can dynamically update word vectors for niche domains.
Summary updated