Short Text Conceptualization Using a Probabilistic Knowledgebase

Yangqiu Song, Haixun Wang, Zhongyuan Wang, Hongsong Li, Weizhu Chen

Introduction

Traditional text mining techniques often struggle with short text snippets, such as tweets or search queries, because they lack the statistical density required for standard algorithms to function effectively. Most existing models rely on the "bag-of-words" approach, which treats text as a collection of tokens and uses statistical co-occurrence to identify latent topics. While this works for long-form documents, it fails on short text due to context sparsity. Furthermore, the "latent topics" generated by models like Latent Dirichlet Allocation (LDA) are often clusters of words that are difficult for humans to interpret. This paper introduces a framework that maps short text to a massive, probabilistic knowledgebase (Probase), enabling machines to perform "human-like" conceptualization—recognizing that terms like "India" and "Brazil" represent "emerging markets" rather than just arbitrary strings.

Methodology: The Probase Framework

The core of this research is the utilization of Probase, a massive probabilistic taxonomy containing over 8 million concepts and 17 million instances harvested from billions of web pages. Unlike deterministic databases such as WordNet, Probase assigns scores to relationships, capturing the "typicality" of an instance for a given concept.

Mapping entities to concepts using a probabilistic taxonomy.

The authors developed a Bayesian inference mechanism to perform three types of conceptualization:

  1. Conceptualizing Instances: Calculating the posterior probability of candidate concepts (e.g., "emerging market") given a set of entities (e.g., "China," "Russia").
  2. Conceptualizing Attributes: Inferring concepts based on mentioned attributes (e.g., "population" and "currency" inferring "country").
  3. The Noisy-OR Model: A discriminative model used to handle terms of unknown types, allowing the system to decide if a word like "apple" refers to a fruit or a technology company based on surrounding context.

Experimental Setup and Datasets

The approach was evaluated using a dataset of 605,501 Twitter messages. To test clustering accuracy, the researchers designed two specific sub-problems:

  • Problem 1 (Unique Concepts): Categorizing tweets into distinct buckets: Tech Companies, Pets, and BRIC countries.
  • Problem 2 (Subtle Differences): A more challenging task grouping tweets into four overlapping geographic categories (e.g., North America vs. SE Asia vs. Europe).

Key Results and Insights

The Probase-based conceptualization significantly outperformed traditional methods in short-text clustering. On the "Unique Concepts" task, the proposed method achieved a clustering purity score of 0.911, vastly superior to LDA (0.561) and WordNet-based methods (0.563).

Probabilistic distribution of concepts inferred from a set of instances.

Key insights from the results include:

  • Typicality and Ranking: Probase recognizes that while a "cat" might technically be a "tracked vehicle" in obscure contexts, it is typically an "animal." This probabilistic ranking filters out the noise that plagues other semantic methods.
  • Granularity: Having millions of concepts allows the system to capture the subtlety of human language, distinguishing between specific categories like "Asian countries" versus "Emerging markets."
  • Interpretability: By mapping text to explicit concepts rather than mathematical latent topics, the results are immediately interpretable by human users.

Limitations and Future Work

Despite its success, the framework has limitations. The system relies on heuristic rules to identify disjoint cliques when handling unrelated concepts in a single set, which may not scale to highly complex text. Additionally, the system is entirely dependent on the quality and breadth of Probase; any biases or gaps in the knowledgebase will reflect in the conceptualization. Finally, mapping terms to millions of potential concepts is computationally intensive, requiring significant optimization. Future work may involve integrating these conceptualization techniques into broader NLP tasks such as sentiment analysis and automated reasoning.

Summary updated

Abstract

Most text mining tasks, including clustering and topic detection, are based on statistical methods that treat text as bags of words. Semantics in the text is largely ignored in the mining process, and mining results often have low interpretability. One particular challenge faced by such approaches lies in short text understanding, as short texts lack enough content from which statistical conclusions can be drawn easily. In this paper, we improve text understanding by using a probabilistic knowledgebase that is as rich as our mental world in terms of the concepts (of worldly facts) it contains. We then develop a Bayesian inference mechanism to conceptualize words and short text. We conducted comprehensive experiments on conceptualizing textual terms, and clustering short pieces of text such as Twitter messages. Compared to purely statistical methods such as latent semantic topic modeling or methods that use existing knowledge-bases (e.g., WordNet, Freebase and Wikipedia), our approach brings significant improvements in short text understanding as reflected by the clustering accuracy.
BibTeX
@inproceedings{song2011short,
  author = {Yangqiu Song and Haixun Wang and Zhongyuan Wang and Hongsong Li and Weizhu Chen},
  title = {{Short Text Conceptualization Using a Probabilistic Knowledgebase}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  doi = {10.5591/978-1-57735-516-8/ijcai11-388},
}
Figure