Hierarchical relational models for document networks

Jonathan Chang, David M. BleiThe Annals of Applied Statistics 2010 · DOI

Abstract

We develop the relational topic model (RTM), a hierarchical model of both network structure and node attributes. We focus on document networks, where the attributes of each document are its words, that is, discrete observations taken from a fixed vocabulary. For each pair of documents, the RTM models their link as a binary random variable that is conditioned on their contents. The model can be used to summarize a network of documents, predict links between them, and predict words within them. We derive efficient inference and estimation algorithms based on variational methods that take advantage of sparsity and scale with the number of links. We evaluate the predictive performance of the RTM for large networks of scientific abstracts, web documents, and geographically tagged news.

Introduction

Document networks—collections of documents connected by links such as citations, hyperlinks, or geographic proximity—are ubiquitous in modern data science. Traditional analysis typically bifurcates these datasets, applying topic models like Latent Dirichlet Allocation (LDA) to the text and network analysis tools to the link structure. However, these separate approaches fail to capture the inherent dependency between what a document says and how it is connected.

A document network visualization illustrating the joint modeling of content and connectivity.

The Relational Topic Model (RTM) addresses this by jointly modeling attributes and structure. By conditioning link probabilities on the latent topics that generate the text, the RTM learns "relational topics" that are predictive of connectivity. This allows for powerful applications such as predicting citations for out-of-sample documents and suggesting keywords based on a document's position in a network.

The Relational Topic Model

The RTM extends the LDA framework. In LDA, each document is modeled as a mixture of latent topics, and each word is drawn from a topic-specific distribution. The RTM introduces a link variable $y_{d,d'}$ for each pair of documents $(d, d')$. This variable is modeled as a binary random variable whose distribution depends on the topic assignments $z_d$ and $z_{d'}$ of the words in the respective documents.

Plate notation for the Relational Topic Model (RTM), depicting the hierarchical relationship between topics ($\theta$), words ($w$), and links ($y$).

The probability of a link is defined via a link function $\psi$, which maps the similarity of topic assignments to a probability. To ensure scalability, the model utilizes a variational inference algorithm that exploits the sparsity of real-world networks. By focusing on observed links during the E-step, the computation scales with the number of links rather than the square of the number of documents ($D^2$).

Experimental Evaluation

The RTM was evaluated on four diverse datasets:

  • Cora: A network of computer science research abstracts and citations.
  • WebKB: Hyperlinked web pages from university computer science departments.
  • PNAS: Abstracts and internal citations from the Proceedings of the National Academy of Sciences.
  • LocalNews: A network of news stories from 50 US states, where links represent geographical adjacency.

Key Results and Insights

The RTM demonstrated superior performance in both link and word prediction. On the Cora dataset, it achieved a nearly 40% improvement in predictive rank over baseline models. A key insight is the model's ability to resolve semantic ambiguities using link structure. For instance, it successfully separated "genetic algorithms" (CS) from "DNA sequencing" (Biology)—two fields with overlapping vocabularies but distinct citation patterns.

Furthermore, on the LocalNews dataset, the RTM identified geographically localized topics that standard LDA missed, demonstrating the value of incorporating spatial structure into topic discovery.

Conclusion

The Relational Topic Model provides a robust framework for the joint analysis of document content and network structure. While currently limited by the bag-of-words assumption and a fixed topic count, it establishes a foundation for more complex relational models. Future research directions include integrating syntactic information and employing non-parametric Bayesian methods for automatic topic discovery.

Summary updated

Abstract

We develop the relational topic model (RTM), a hierarchical model of both network structure and node attributes. We focus on document networks, where the attributes of each document are its words, that is, discrete observations taken from a fixed vocabulary. For each pair of documents, the RTM models their link as a binary random variable that is conditioned on their contents. The model can be used to summarize a network of documents, predict links between them, and predict words within them. We derive efficient inference and estimation algorithms based on variational methods that take advantage of sparsity and scale with the number of links. We evaluate the predictive performance of the RTM for large networks of scientific abstracts, web documents, and geographically tagged news.
BibTeX
@article{chang2010hierarchical,
  author = {Jonathan Chang and David M. Blei},
  title = {{Hierarchical relational models for document networks}},
  journal = {The Annals of Applied Statistics},
  year = {2010},
  doi = {10.1214/09-aoas309},
}
Figure