Sparse Relational Topic Models for Document Networks

Aonan Zhang, Jun Zhu, Bo ZhangECML/PKDD 2013 · OpenReview

Abstract

Learning latent representations is playing a pivotal role in machine learning and many application areas. Previous work on relational topic models (RTM) has shown promise on learning latent topical representations for describing relational document networks and predicting pairwise links. However under a probabilistic formulation with normalization constraints, RTM could be ineffective in controlling the sparsity of the topical representations, and may often need to make strict mean-field assumptions for approximate inference. This paper presents sparse relational topic models (SRTM) under a non-probabilistic formulation that can effectively control the sparsity via a sparsity-inducing regularizer. Our model can also handle imbalance issues in real networks via introducing various cost parameters for positive and negative links. The deterministic optimization problem of SRTM admits efficient coordinate descent algorithms. We also present a generalization to consider all pairwise topic interactions. Our empirical results on several real network datasets demonstrate better performance on link prediction, sparser latent representations, and faster running time than the competitors under a probabilistic formulation.

Introduction

Predicting links between documents—such as which paper will cite another or which webpage will link to a friend—requires understanding both the network structure and the content of the documents. The gold standard has been the Relational Topic Model (RTM). However, RTM suffers from three major "real-world" headaches: difficulty in controlling sparsity due to probabilistic normalization constraints, struggles with extreme data imbalance (where links often represent <0.1% of possible pairs), and high computational complexity stemming from restrictive mean-field inference assumptions.

To address these limitations, we introduce the Sparse Relational Topic Model (SRTM). By moving away from traditional probabilistic constraints toward a non-probabilistic formulation with $\ell_1$ regularization, SRTM provides a more flexible and efficient way to predict links in document networks.

Methodology

SRTM is built upon Sparse Topical Coding (STC) rather than Latent Dirichlet Allocation (LDA). This shift allows the model to treat topic assignments as codes that can be directly regularized for sparsity.

Graphical representation of the Relational Topic Model framework showing dependencies between document topics (theta), word codes (s), observed words (w), and document links (y).

Key Technical Innovations

  • Non-probabilistic Framework: By relaxing the requirement that topic distributions must sum to one, we apply an $\ell_1$-norm regularizer. This directly induces sparsity in the word codes ($s_{dn}$), ensuring each word is associated only with its most relevant topics.
  • Cost-Sensitive Learning: We introduce specific weight parameters ($C_+$ and $C_-$) for positive and negative links. This allows the model to "pay more attention" to rare positive links, effectively solving the data imbalance problem.
  • Coordinate Descent Optimization: The model's objective function is bi-convex, allowing for an efficient deterministic optimization algorithm rather than complex probabilistic inference. This makes the model significantly faster and more scalable.
  • Generalized SRTM (gSRTM): We also propose a version that uses a full weight matrix instead of a diagonal one. This allows the model to capture interactions between different topics (e.g., a "Statistical Theory" paper citing a "Machine Learning" paper), rather than just looking for topic overlap.

Experimental Setup

The model was evaluated on three classic document network benchmarks characterized by extreme link sparsity:

  • Cora: 2,708 research papers; 5,429 links (0.07% link density).
  • WebKB: 877 webpages; 1,703 links (0.2% link density).
  • CiteSeer: 3,312 papers; 4,714 links (0.04% link density).

Key Results and Insights

  • Superior Prediction: SRTM/gSRTM consistently outperformed RTM and two-step baselines. On the Cora dataset, SRTM improved the "link rank" accuracy by approximately 100 points over RTM.
  • Efficiency and Scalability: SRTM is roughly 2x faster to train than RTM when the number of topics is large, thanks to the shift from inference to deterministic optimization.
  • True Sparsity and Interpretability: SRTM achieved a word-code sparsity ratio of nearly 90% while maintaining high accuracy. This makes the latent representations much cleaner and more interpretable than those produced by RTM, where accuracy typically plummets if forced toward sparsity.
  • Beyond Topic Overlap: The generalized model (gSRTM) demonstrated that papers with different but related topics are often more likely to link than those with identical topics, a nuance that standard RTM-style models often miss.

Conclusion

SRTM demonstrates that a non-probabilistic approach to relational topic modeling can yield significant gains in both performance and efficiency. By inducing sparsity and accounting for link imbalance, SRTM provides a robust framework for analyzing complex document networks. Future work will explore stochastic gradient descent to scale the model to massive, ever-growing networks and investigate automated methods for hyperparameter estimation.

Summary updated

Abstract

Learning latent representations is playing a pivotal role in machine learning and many application areas. Previous work on relational topic models (RTM) has shown promise on learning latent topical representations for describing relational document networks and predicting pairwise links. However under a probabilistic formulation with normalization constraints, RTM could be ineffective in controlling the sparsity of the topical representations, and may often need to make strict mean-field assumptions for approximate inference. This paper presents sparse relational topic models (SRTM) under a non-probabilistic formulation that can effectively control the sparsity via a sparsity-inducing regularizer. Our model can also handle imbalance issues in real networks via introducing various cost parameters for positive and negative links. The deterministic optimization problem of SRTM admits efficient coordinate descent algorithms. We also present a generalization to consider all pairwise topic interactions. Our empirical results on several real network datasets demonstrate better performance on link prediction, sparser latent representations, and faster running time than the competitors under a probabilistic formulation.
BibTeX
@inproceedings{zhang2013sparse,
  author = {Aonan Zhang and Jun Zhu and Bo Zhang},
  title = {{Sparse Relational Topic Models for Document Networks}},
  booktitle = {ECML/PKDD},
  year = {2013},
  url = {https://openreview.net/forum?id=Stbq50j8jrg},
}
Figure