Less is More: Learning Prominent and Diverse Topics for Data Summarization

Jian Tang, Ming Zhang, Qiaozhu MeiarXiv 2016 · OpenReview

TL;DR

This paper introduces "diverse topic models" (DivPLSA and DivLDA) that learn fewer, more representative topics for data summarization. By employing a novel "reinforced random walk" mechanism, prominent topics absorb information from similar, smaller topics, leading to more diverse and informative summaries with less user input.

What Problem Does This Paper Address?

In the age of "big data," understanding and exploring vast amounts of unstructured text is a significant challenge. While statistical topic models like Latent Dirichlet Allocation (LDA) are powerful tools for uncovering hidden themes, they typically require users to specify the exact number of topics beforehand. This is problematic because users often lack prior knowledge of the data's true topic structure. Furthermore, existing methods that try to relax this constraint often lead to overly complex models or topics of compromised quality. The core problem is how to efficiently extract a small, representative set of topics that accurately summarize a large dataset without demanding extensive prior knowledge from the user.

Methods

The paper's main innovation is a "reinforced random walk" process integrated into classical topic models. This process operates on a "social network" of topics, where topics are nodes and edges represent their similarity (cosine similarity of word distributions). When a word token is assigned to a topic, it can "walk" to other topics. The probability of this walk is reinforced by the size of the target topic. This mechanism encourages larger, more prominent topics to absorb tokens from smaller, similar topics, effectively merging them. This process naturally leads to fewer, more dominant topics that are both representative and diverse. The authors embed this reinforced random walk into the inference steps of Probabilistic Latent Semantic Analysis (PLSA) and Latent Dirichlet Allocation (LDA), resulting in two new models: DivPLSA and DivLDA. The inference procedures remain computationally efficient, similar to their classical counterparts.

Datasets

The authors evaluated their models on four real-world datasets:

  • 4CONF: A small dataset of conference papers, useful for intuitive topic interpretation.
  • 20NG: A subset of the widely used 20 newsgroups dataset.
  • WIKIPEDIA: A large dataset of Wikipedia articles.
  • DBLP: A comprehensive dataset of computer science bibliography abstracts.

These datasets are standard benchmarks in the field, allowing for direct comparison with existing methods.

Key Insights and Ideas

  • "Less is More" for Summarization: The paper argues that for data exploration and summarization, users often need a few high-quality topics rather than an exhaustive list. This challenges the traditional topic modeling paradigm of fitting a fixed, often large, number of topics.
  • Reinforced Random Walk for Topic Merging: The core idea of allowing prominent topics to "absorb" smaller, similar ones is a clever way to achieve topic reduction and diversity simultaneously. It mimics a "rich get richer" phenomenon for topics.
  • Reduced User Burden: By allowing the model to learn the "right" number of prominent topics, the need for users to guess the optimal number of topics is significantly reduced, making topic modeling more accessible.
  • Balancing Prominence and Diversity: The reinforced random walk mechanism inherently balances these two crucial aspects of a good summary. Prominent topics emerge due to absorption, and diversity is enhanced because topics tend to represent different "neighborhoods" in the topic network.

Key Results

  • Improved Summarization: DivPLSA and DivLDA consistently outperformed classical PLSA/LDA and their "TopK" variants (selecting largest topics) in terms of both semantic coherence (PMI) and information coverage (perplexity) on the evaluated datasets.
  • Reduced Topic Count: The models naturally converged to a smaller number of active topics (e.g., 5 topics from an initial 20 in the 4CONF dataset), demonstrating their ability to effectively merge redundant topics.
  • Robustness to Parameters: The performance of DivPLSA and DivLDA was found to be relatively insensitive to the parameters controlling reinforcement intensity ($\gamma$) and the initial number of topics ($K$), simplifying their application.
  • Qualitative Success: For the 4CONF dataset, the learned topics provided a good summary, identifying key areas like "information retrieval," "web," "data mining," "neural network," and "learning, Bayesian."

Limitations and Future Work

  • Theoretical Convergence: The authors acknowledge that the theoretical justification for the convergence of their models (which lack explicit objective functions after incorporating the random walk) is left as future work. They empirically demonstrate convergence through likelihood and active topic counts.
  • Generalization to Other Tasks: While effective for summarization, the authors suggest applying the reinforced random walk mechanism to other scenarios, such as unsupervised clustering, to learn prominent and diverse clusters.
  • Parameter Optimization: While performance is not highly sensitive, the paper notes that DivLDA tends to use a smaller $\gamma$ than DivPLSA, possibly due to its Gibbs sampling nature and hyperparameter optimization. Further investigation into optimal parameter settings could be beneficial.

Summary updated

Abstract

Statistical topic models efficiently facilitate the exploration of large-scale data sets. Many models have been developed and broadly used to summarize the semantic structure in news, science, social media, and digital humanities. However, a common and practical objective in data exploration tasks is not to enumerate all existing topics, but to quickly extract representative ones that broadly cover the content of the corpus, i.e., a few topics that serve as a good summary of the data. Most existing topic models fit exactly the same number of topics as a user specifies, which have imposed an unnecessary burden to the users who have limited prior knowledge. We instead propose new models that are able to learn fewer but more representative topics for the purpose of data summarization. We propose a reinforced random walk that allows prominent topics to absorb tokens from similar and smaller topics, thus enhances the diversity among the top topics extracted. With this reinforced random walk as a general process embedded in classical topic models, we obtain \textit{diverse topic models} that are able to extract the most prominent and diverse topics from data. The inference procedures of these diverse topic models remain as simple and efficient as the classical models. Experimental results demonstrate that the diverse topic models not only discover topics that better summarize the data, but also require minimal prior knowledge of the users.
BibTeX
@misc{tang2016less,
  author = {Jian Tang and Ming Zhang and Qiaozhu Mei},
  title = {{Less is More: Learning Prominent and Diverse Topics for Data Summarization}},
  year = {2016},
  eprint = {1611.09921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.09921},
}
Figure