Clustering via Content-Augmented Stochastic Blockmodels

J. Massey Cashore, Xiaoting Zhao, Alexander A. Alemi, Yujia Liu, Peter I. FrazierarXiv 2015 · arXiv

Abstract

Community detection in bipartite networks often ignores the rich content of the items being interacted with, while text-only clustering ignores the powerful signal of user behavior. We present the Content-Augmented Stochastic Blockmodel (CASB), a unified generative model that integrates bipartite graph interactions with a probabilistic model of item text. By leveraging both signals, CASB achieves superior performance in identifying specialized research sub-communities on platforms like arXiv and INFORMS, effectively solving the "cold-start" problem for new items and providing a robust regularizer against interaction noise.

Introduction

Identifying "communities" or "clusters" in data is a cornerstone of modern recommendation systems and information filtering. Traditionally, researchers employ Stochastic Blockmodels (SBMs) to find patterns in how users interact with items—for instance, which scientists read which papers. However, these models often treat items as "black boxes," ignoring the rich information contained within the items themselves, such as the text of a scientific abstract. Conversely, text-only clustering ignores user behavior, which can be a powerful signal of intent and expertise.

This paper addresses this gap by introducing CASB (Content-Augmented Stochastic Blockmodels). The primary motivation is the arXiv preprint server, where broad categories (like "Cosmology") are often too coarse for researchers who require highly specific daily feeds tailored to their sub-niche. By simultaneously analyzing who interacts with what and the actual content of those items, CASB improves community detection accuracy and interpretability.

The CASB Model

The CASB model treats the problem as a bipartite graph where users and items belong to latent (hidden) communities and clusters. The model consists of two primary components:

  1. The Interaction Component: This assumes the probability of a user $j$ finding item $i$ relevant depends entirely on their respective group memberships, following a standard bipartite SBM structure.
  2. The Content Component: This is the model's core innovation. Each item cluster is associated with a unique probability distribution over words (a Multinomial distribution). If an item belongs to a specific cluster, its text is assumed to be generated from that cluster's specific "vocabulary profile."
The CASB generative model structure illustrating the dependencies between latent communities, interactions, and content.

Because calculating the exact relationship between all these variables is mathematically intractable, the authors utilize Mean-Field Variational Inference. This approach allows for the efficient approximation of hidden cluster assignments by optimizing a lower bound on the model's likelihood.

Experimental Evaluation

The model was evaluated on three real-world datasets:

  • arXiv astro-ph.CO (Cosmology): 7,819 papers and 621 frequent users.
  • arXiv hep-th (Theoretical High Energy Physics): 6,677 papers and 1,449 frequent users.
  • INFORMS Conference Data: Five years of conference proceedings where authors are treated as "users" and their co-presenters/papers form the interaction network.

Key Results and Insights

Solving the "Misplaced Paper" Mystery

A compelling application of CASB involved a historical quirk on arXiv where "Galactic" and "Nongalactic" astrophysics papers were often lumped into the same category due to conflicting naming conventions. CASB successfully separated these two distinct research communities in an unsupervised manner. In the task of separating "Cosmology" from "Galaxy" papers, CASB using a bag-of-words representation achieved the best performance, with only 884 misplaced papers, significantly beating benchmarks like KMeans on LDA (936) and Link-PLSA-LDA (985).

Content as a Regularizer

Interaction data can be noisy; users may click on items by accident or out of curiosity. By forcing the model to explain the content of the items, the text acts as a "sanity check" that prevents the model from creating clusters based on random interaction noise. Furthermore, because the model understands content, it can address the cold-start problem, assigning a brand-new paper (with zero views) to a cluster based solely on its text.

Qualitative Success

On the INFORMS dataset, CASB successfully identified distinct, recognizable research communities. As shown in the word clouds learned by the model, it clearly distinguished between "Optimization" and "Transportation Logistics" themes based on the vocabulary profiles of the clusters.

Word clouds for Optimization and Transportation clusters generated from the learned CASB parameters.

Conclusion

CASB demonstrates that augmenting structural interaction data with content information leads to more robust and interpretable community detection. While the current model focuses on binary feedback and requires vocabulary truncation for scalability, it provides a strong foundation for future work. Potential extensions include incorporating more nuanced feedback (like ratings or dwell time) and developing dynamic models to account for the evolution of research interests over time.

Summary updated

Abstract

Much of the data being created on the web contains interactions between users and items. Stochastic blockmodels, and other methods for community detection and clustering of bipartite graphs, can infer latent user communities and latent item clusters from this interaction data. These methods, however, typically ignore the items' contents and the information they provide about item clusters, despite the tendency of items in the same latent cluster to share commonalities in content. We introduce content-augmented stochastic blockmodels (CASB), which use item content together with user-item interaction data to enhance the user communities and item clusters learned. Comparisons to several state-of-the-art benchmark methods, on datasets arising from scientists interacting with scientific articles, show that content-augmented stochastic blockmodels provide highly accurate clusters with respect to metrics representative of the underlying community structure.
BibTeX
@misc{cashore2015clustering,
  author = {J. Massey Cashore and Xiaoting Zhao and Alexander A. Alemi and Yujia Liu and Peter I. Frazier},
  title = {{Clustering via Content-Augmented Stochastic Blockmodels}},
  year = {2015},
  eprint = {1505.06538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1505.06538},
}
Figure