Abstract
The topic model has been widely applied to various research domains such as information retrieval and data mining to discover topics in an unsupervised way. While early research focused on long texts, the emergence of the Internet has led to a rapid growth in short-text data. Existing schemes to solve the sparsity problems of short texts often rely on data aggregation or model improvements, such as the Biterm Topic Model (BTM). However, these strategies often ignore semantically similar but rarely co-occurrent word pairs and the multi-sense phenomenon (polysemy) in natural languages. In this paper, we introduce MuSeS, a novel short-text topic model that utilizes multi-sense word vectors to extract similar word pairs from the whole corpus. By disambiguating multiple senses of words and generating more reasonable global biterms, MuSeS achieves superior results on open-source English datasets compared to state-of-the-art models.
Introduction
Topic modeling is a staple of NLP, but it struggles with "short texts" like tweets or news headlines. Traditional models like Latent Dirichlet Allocation (LDA) rely on word co-occurrence within a single document; in a five-word tweet, there simply isn't enough context to build a reliable model. This is known as the sparsity problem.
The Biterm Topic Model (BTM) improved this by modeling word pairs (biterms) across the entire corpus. However, two major gaps remain:
- Semantic Blindness: BTM only considers words that actually appear together. If "essay" and "dissertation" are synonyms but never appear in the same document, BTM won't link them.
- Polysemy: Most models treat a word like "apple" the same whether it refers to a fruit or a tech giant. This "sense-mixing" creates noisy, overlapping topics.
Methodology: The MuSeS Framework
The proposed MuSeS (Multiple Senses Short-text Topic Model) addresses these gaps through a three-phase process designed to densify sparse data and handle polysemy.
1. Sense Annotation
Instead of using static word vectors, MuSeS utilizes contextualized embeddings (such as ELMo). Every instance of a word in the corpus receives a unique vector based on its surrounding context. The model clusters these vectors to identify distinct "senses" (e.g., apple#0 vs. apple#1) and re-annotates the entire corpus with these sense-specific tags.
2. Biterm Generation
The model generates two distinct sets of biterms:
- Local Biterms: Standard co-occurrences within a single short-text document.
- Global Biterms: The model looks at the entire corpus to find words that are semantically similar based on their sense vectors. It creates biterms for these pairs even if they never co-occur locally, effectively "densifying" the sparse dataset.
3. Topic Discovery
MuSeS uses a modified Gibbs sampling procedure to learn topic distributions over these sense-aware biterms. This allows the model to "attend" to the specific meaning of a word in its given context, preventing the "topic bleed" that occurs when a polysemous word acts as a bridge between unrelated topics.
Experimental Setup and Results
The researchers evaluated MuSeS on two primary English datasets: Online Questions (50,253 titles from Yahoo! Answers) and Online News (172,546 headlines from the UCI Machine Learning Repository). MuSeS was compared against four baselines: LDA, BTM, GPU-DMM, and GloSS.
Key Findings
- Topic Coherence: MuSeS achieved the highest coherence scores across almost all test cases, indicating that the discovered topics are more human-interpretable and logically consistent.
- Classification Accuracy: On the Online Questions dataset, MuSeS achieved a classification accuracy of 50.51% (using a Random Forest classifier), significantly higher than BTM (40.49%) and the previous state-of-the-art, GloSS (49.37%).
- Clustering: MuSeS showed superior performance in Purity and Homogeneity metrics, proving it creates more distinct and accurate document groupings.
- Extensibility: While the paper primarily uses ELMo, the framework is "pluggable." Swapping ELMo for BERT further improved performance, suggesting the model scales well with advancing embedding technologies.
Conclusion
By moving beyond document-level co-occurrence to corpus-level semantic similarity and addressing word sense disambiguation, MuSeS provides a robust solution for short-text topic discovery. While the current model is designed for static datasets, future work will focus on adapting MuSeS for streaming texts, where topics and word senses may shift dynamically over time.
Summary updated