Polysemy Needs Attention: Short-Text Topic Discovery With Global and Multi-Sense Information

Heng-Yang Lu, Jun Yang, Yi Zhang, Zuoyong LiIEEE Access 2021 · OpenReview

Abstract

The topic model has been widely applied to various research domains such as information retrieval and data mining to discover topics in an unsupervised way. While early research focused on long texts, the emergence of the Internet has led to a rapid growth in short-text data. Existing schemes to solve the sparsity problems of short texts often rely on data aggregation or model improvements, such as the Biterm Topic Model (BTM). However, these strategies often ignore semantically similar but rarely co-occurrent word pairs and the multi-sense phenomenon (polysemy) in natural languages. In this paper, we introduce MuSeS, a novel short-text topic model that utilizes multi-sense word vectors to extract similar word pairs from the whole corpus. By disambiguating multiple senses of words and generating more reasonable global biterms, MuSeS achieves superior results on open-source English datasets compared to state-of-the-art models.

Introduction

Topic modeling is a staple of NLP, but it struggles with "short texts" like tweets or news headlines. Traditional models like Latent Dirichlet Allocation (LDA) rely on word co-occurrence within a single document; in a five-word tweet, there simply isn't enough context to build a reliable model. This is known as the sparsity problem.

The Biterm Topic Model (BTM) improved this by modeling word pairs (biterms) across the entire corpus. However, two major gaps remain:

  1. Semantic Blindness: BTM only considers words that actually appear together. If "essay" and "dissertation" are synonyms but never appear in the same document, BTM won't link them.
  2. Polysemy: Most models treat a word like "apple" the same whether it refers to a fruit or a tech giant. This "sense-mixing" creates noisy, overlapping topics.

Methodology: The MuSeS Framework

The proposed MuSeS (Multiple Senses Short-text Topic Model) addresses these gaps through a three-phase process designed to densify sparse data and handle polysemy.

The MuSeS framework architecture, illustrating the pipeline from embedding learning to topic inference.

1. Sense Annotation

Instead of using static word vectors, MuSeS utilizes contextualized embeddings (such as ELMo). Every instance of a word in the corpus receives a unique vector based on its surrounding context. The model clusters these vectors to identify distinct "senses" (e.g., apple#0 vs. apple#1) and re-annotates the entire corpus with these sense-specific tags.

2. Biterm Generation

The model generates two distinct sets of biterms:

  • Local Biterms: Standard co-occurrences within a single short-text document.
  • Global Biterms: The model looks at the entire corpus to find words that are semantically similar based on their sense vectors. It creates biterms for these pairs even if they never co-occur locally, effectively "densifying" the sparse dataset.

3. Topic Discovery

MuSeS uses a modified Gibbs sampling procedure to learn topic distributions over these sense-aware biterms. This allows the model to "attend" to the specific meaning of a word in its given context, preventing the "topic bleed" that occurs when a polysemous word acts as a bridge between unrelated topics.

Experimental Setup and Results

The researchers evaluated MuSeS on two primary English datasets: Online Questions (50,253 titles from Yahoo! Answers) and Online News (172,546 headlines from the UCI Machine Learning Repository). MuSeS was compared against four baselines: LDA, BTM, GPU-DMM, and GloSS.

Key Findings

  • Topic Coherence: MuSeS achieved the highest coherence scores across almost all test cases, indicating that the discovered topics are more human-interpretable and logically consistent.
  • Classification Accuracy: On the Online Questions dataset, MuSeS achieved a classification accuracy of 50.51% (using a Random Forest classifier), significantly higher than BTM (40.49%) and the previous state-of-the-art, GloSS (49.37%).
  • Clustering: MuSeS showed superior performance in Purity and Homogeneity metrics, proving it creates more distinct and accurate document groupings.
  • Extensibility: While the paper primarily uses ELMo, the framework is "pluggable." Swapping ELMo for BERT further improved performance, suggesting the model scales well with advancing embedding technologies.

Conclusion

By moving beyond document-level co-occurrence to corpus-level semantic similarity and addressing word sense disambiguation, MuSeS provides a robust solution for short-text topic discovery. While the current model is designed for static datasets, future work will focus on adapting MuSeS for streaming texts, where topics and word senses may shift dynamically over time.

Summary updated

Abstract

The topic model has been widely applied to various research domains such as information retrieval, data mining, and so on. It can discover topics of texts in an unsupervised way. In the early years, most researches mainly focused on long texts. With the emergence of the Internet, the number of short texts is growing rapidly. Most existing schemes to solve the sparsity problems of short texts, are mainly based on data aggregation or model improvements. Among them, the Biterm Topic Model is one of the most representative models. It proposed a new way to model topics based on document-level word pairs and has shown creativity and effectiveness. However, this strategy ignores those semantically similar and rarely co-occurrent word pairs. What’s more, most researches ignore the multi-sense phenomenon in natural languages. In this paper, we utilize multi-sense word vectors to extract similar word pairs from the whole corpus by considering multiple senses. Based on this idea, we introduce a novel short-text topic model, which disambiguates multiple senses of words and generates more reasonable global biterms. Experimental results on two open-source English datasets have shown superiority to state-of-the-art topic models.
BibTeX
@inproceedings{lu2021polysemy,
  author = {Heng-Yang Lu and Jun Yang and Yi Zhang and Zuoyong Li},
  title = {{Polysemy Needs Attention: Short-Text Topic Discovery With Global and Multi-Sense Information}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=H7NwOQzv7Be},
}
Figure