Short Text Understanding by Leveraging Knowledge into Topic Model

Shansong Yang, Weiming Lu, Dezhi Yang, Liang Yao, Baogang WeiNAACL 2015 · DOI

Introduction

Traditional topic models like Latent Dirichlet Allocation (LDA) often struggle with short texts, such as tweets or product reviews, due to data sparsity. In these contexts, the lack of sufficient word co-occurrence statistics makes it difficult for models to identify meaningful patterns. This paper introduces the Phrase Topic Model (PTM), which addresses this sparsity problem by automatically identifying key-phrases and using them as semantic anchors to guide the topic discovery process.

The Phrase Topic Model (PTM)

The core innovation of PTM is the treatment of phrases as the primary "key points of interest" in a short text. By leveraging self-contained knowledge found within the text itself, the model avoids the noise often associated with external knowledge bases.

Methodology

The PTM follows a structured generative process:

  1. Phrase Extraction: A Conditional Random Field (CRF++) tool is used to identify key-phrases (e.g., "USB car adapter") from the raw text.
  2. Generative Process: Unlike standard LDA, PTM enforces that all words within a single key-phrase belong to the same topic. These phrases act as anchors; for each document, the model first assigns topics to the key-phrases. Non-phrase words then draw their topic assignments from the mixture of topics already established by the phrases in that specific document.
  3. Inference: The model uses Collapsed Gibbs Sampling to estimate hidden topic assignments, allowing it to learn cohesive word and phrase groupings across the entire corpus.
Graphical model of the Phrase Topic Model (PTM) showing the plate notation and the dependency of words on phrase-based topic assignments.

Experimental Evaluation

The researchers evaluated PTM using an Online Reviews Dataset across four domains: Computer, Cellphone, Camera, and Food. Each domain contained 500 reviews with an average length of approximately 20 words, representing a classic short-text challenge.

Key Results and Insights

PTM was compared against several baselines, including LDA, the Biterm Topic Model (BTM), and external knowledge-based models (GK-LDA and DF-LDA).

  • Superior Topic Coherence: PTM consistently achieved higher topic coherence scores than all baseline models. This indicates that the top words in PTM-generated topics are more semantically related.
  • Stability Across Topic Scales: As illustrated in the performance metrics, PTM maintains stable and superior performance regardless of the number of topics (K) chosen, whereas LDA's performance is significantly less consistent on short texts.
  • Qualitative Improvements: Side-by-side comparisons (such as Table 3 in the original paper) show that PTM produces much cleaner topics. For instance, in a "Music" topic for cellphones, PTM correctly identified terms like "radio" and "sound," while LDA included unrelated noise like "hands" and "charge."
Topic Coherence vs. Number of Topics, demonstrating that PTM (OurModel) provides more coherent and stable results compared to LDA and other baselines.

Conclusion

By treating phrases as semantic units and anchors, the Phrase Topic Model effectively overcomes the sparsity issues inherent in short text topic modeling. The results demonstrate that mining "self-contained" knowledge from the text itself can be more effective than relying on general external knowledge bases. Future research may explore hybrid models that combine these internal phrase anchors with external semantic networks for even more robust short text understanding.

Summary updated

BibTeX
@inproceedings{yang2015short,
  author = {Shansong Yang and Weiming Lu and Dezhi Yang and Liang Yao and Baogang Wei},
  title = {{Short Text Understanding by Leveraging Knowledge into Topic Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1131},
}
Figure