Authors: Rajesh Ranganath and David M. Blei
Abstract
We develop correlated random measures, random measures where the atom weights can exhibit a flexible pattern of dependence, and use them to develop powerful hierarchical Bayesian nonparametric models. Hierarchical Bayesian nonparametric models are usually built from completely random measures, a Poisson-process-based construction in which the atom weights are independent. Completely random measures imply strong independence assumptions in the corresponding hierarchical model, and these assumptions are often misplaced in real-world settings. Correlated random measures address this limitation. They model correlation within the measure by using a Gaussian process in concert with the Poisson process. With correlated random measures, for example, we can develop a latent feature model for which we can infer both the properties of the latent features and their dependency pattern. We develop several other examples as well. We study a correlated random measure model of pairwise count data. We derive an efficient variational inference algorithm and show improved predictive performance on large datasets of documents, web clicks, and electronic health records.
Introduction
In the field of Bayesian Nonparametrics (BNP), researchers often rely on models like the Hierarchical Dirichlet Process (HDP) to discover an unknown number of latent components within data—such as topics in a corpus of text or disease clusters in electronic health records. Traditionally, these models are built upon Completely Random Measures (CRMs).
The term "completely" refers to a specific mathematical property: the weights assigned to the discovered components are independent. In a medical context, a CRM-based model would assume that the probability of a patient being diagnosed with Type 2 Diabetes provides no information about the probability of them also having Heart Failure. In reality, features are almost always correlated. By ignoring these dependencies, standard BNP models sacrifice predictive accuracy and fail to capture the underlying structure of the data. This paper introduces Correlated Random Measures (CorrRMs), a framework that integrates Gaussian processes with Poisson processes to model these essential dependencies.
The Framework: Correlated Random Measures
The core innovation of CorrRMs is the ability to bake correlation directly into the measure's construction. The authors propose a multi-step "recipe" for generating these measures:
- Atom Generation: A Poisson process is used to generate a set of "atoms" (representing features or components) and their initial base weights.
- Latent Geography: Each atom is assigned a location in a latent $d$-dimensional space.
- The Correlation Engine: A random function is drawn from a Gaussian Process (GP) over that latent space.
- Transformation: The final weight of each feature is determined by a transformation of its base weight and the value of the GP at its specific latent location.
This approach creates a "topology" of features. If two features are positioned close together in the latent space, the GP will likely assign them similar values, causing their weights to covary. This allows the model to simultaneously learn the features themselves and the map of how they relate to one another.
Scalable Inference via CNPF
To demonstrate the utility of this framework, the authors developed the Correlated Nonparametric Poisson Factorization (CNPF) model, specifically designed for large, sparse count matrices.
A common bottleneck in Gaussian Process models is the $O(N^3)$ complexity associated with matrix inversions. To ensure scalability, the authors utilized a linear kernel and an auxiliary variable representation. This allowed them to implement a Stochastic Variational Inference (SVI) algorithm that can handle massive datasets without the prohibitive computational costs typically associated with GPs.
Empirical Evaluation
The authors evaluated the CorrRM framework across three diverse, large-scale datasets:
- The New York Times: 100,000 articles used for topic modeling.
- Mayo Clinic: Electronic health records for over 142,000 patients across 11,102 diagnostic codes.
- ArXiv Click Data: A recommendation task involving 50,000 users and 20,000 articles.
Key Results
Across all three domains, the correlated models (CNPF) significantly outperformed uncorrelated baselines in terms of held-out perplexity, a standard metric for how well a model predicts unseen data.
Beyond predictive power, the model provided richer qualitative insights:
- Medical Discovery: In the Mayo Clinic dataset, the model automatically identified clinically relevant correlations, such as the links between obesity and Type 2 Diabetes, or depression and alcohol dependency.
- Information Topology: In the New York Times data, the model recovered a map of topics where "international politics" acted as a bridge between "finance" and "law" clusters.
- Adaptive Complexity: The model demonstrated its nonparametric nature by identifying that while 200 potential components were available, only 50–75 were significantly utilized to explain the data.
Conclusion and Future Directions
Correlated Random Measures represent a significant advancement in BNP, providing a unified framework to create correlated versions of the Gamma, Beta, and Dirichlet processes. While the current work utilizes linear kernels for efficiency, future research could explore more complex kernels (like RBF) using "inducing variables" to capture even more intricate dependencies. Additionally, the framework is flexible enough to support other stochastic processes, such as Student-t processes or Brownian motion, opening new avenues for modeling complex, dependent latent structures in high-dimensional data.
Summary updated