Correlated Random Measures

Rajesh Ranganath, David M. Blei

This paper, "Correlated Random Measures" by Rajesh Ranganath and David M. Blei, introduces a significant upgrade to the toolbox of Bayesian Nonparametrics (BNP). It moves beyond the restrictive assumption that latent features in a model must be independent, providing a mathematically elegant way to bake real-world correlations into complex data models.

TL;DR

Most Bayesian nonparametric models assume that the latent features underlying data are independent, an assumption that rarely holds in reality. This paper introduces Correlated Random Measures (CorrRMs), a framework that uses Gaussian processes to model dependencies between features, leading to significantly better predictive performance and richer, more interpretable latent structures.

What Problem Does This Paper Address?

In the world of Bayesian nonparametrics, we often use models like the Hierarchical Dirichlet Process (HDP) to discover an unknown number of "components" in data—think of these as topics in a collection of documents or disease clusters in medical records.

Standard versions of these models are built on Completely Random Measures (CRMs). The "completely" here is a technical term meaning that the weights of the discovered components are independent. For example, in a medical model, a CRM assumes that the probability of a patient having "Type 2 Diabetes" tells you absolutely nothing about the probability of them having "Heart Failure."

In reality, features are almost always correlated. Ignoring these dependencies makes models less accurate and misses the "big picture" of how different components relate to one another.

Methods

The authors propose Correlated Random Measures (CorrRMs). Here is the high-level recipe:

  1. Atom Generation: Use a Poisson process to generate a set of "atoms" (the features) and their base weights.
  2. Latent Geography: Assign each atom a location in a latent $d$-dimensional space.
  3. The Correlation Engine: Draw a random function from a Gaussian Process (GP) over that latent space.
  4. Transformation: The final weight of a feature is a transformation of its base weight and the value of the GP at its latent location.

If two features are close together in this latent space, the GP will assign them similar values, causing their weights to covary. This allows the model to learn both the features themselves and the "map" of how they relate.

To make this practical, the authors developed a scalable Stochastic Variational Inference (SVI) algorithm. They specifically focus on a model called Correlated Nonparametric Poisson Factorization (CNPF), which is designed for large, sparse count matrices.

Datasets

The authors evaluated their model on three large-scale, real-world datasets:

  • The New York Times: 100,000 articles and an 8,000-word vocabulary used for topic modeling.
  • Mayo Clinic: Electronic health records for 142,297 patients across 11,102 diagnostic codes.
  • ArXiv Click Data: A recommendation task involving 50,000 users and 20,000 articles.

Key Insights and Ideas

  • Proximity equals Correlation: By placing features in a latent space, the model recovers a "topology" of information. In the NYT data, the model discovered that "international politics" serves as a bridge (or "glue") between the "finance" and "law" clusters.
  • A Unified Framework: CorrRMs aren't just one model; they are a construction. The authors show how this can be used to create correlated versions of the Gamma process, the Beta process, and the Dirichlet process.
  • Efficiency via Linear Kernels: To avoid the $O(N^3)$ complexity typical of Gaussian Processes, the authors used a linear kernel and an auxiliary variable representation, allowing them to evaluate likelihoods without expensive matrix inversions.

Key Results

  • Superior Prediction: On all three datasets, the correlated models (CNPF) significantly outperformed uncorrelated baselines in terms of held-out perplexity (a measure of how well the model predicts unseen data).
  • Meaningful Medical Discovery: In the Mayo Clinic data, the model automatically identified well-known medical correlations, such as the link between obesity and Type 2 Diabetes, and depression and alcohol dependency.
  • Sparsity Handling: The model successfully identified that while there were 200 potential components available (the truncation level), only about 50–75 were heavily used, demonstrating the "nonparametric" ability to adapt to the data's complexity.

Limitations and Future Work

  • Kernel Complexity: While the linear kernel is efficient, more complex kernels (like RBF) might capture even richer dependencies but would require more sophisticated inference techniques like "inducing variables" to remain scalable.
  • Theoretical Rates: The authors suggest that future work should investigate the theoretical convergence rates of CorrRMs compared to standard CRMs to understand exactly how much more "data efficient" they are.
  • Alternative Processes: While this paper uses Gaussian Processes, the framework is flexible enough that one could theoretically swap the GP for other stochastic processes, such as Student-t processes or Brownian motion, depending on the desired correlation structure.

Summary updated

Abstract

We develop correlated random measures, random measures where the atom weights can exhibit a flexible pattern of dependence, and use them to develop powerful hierarchical Bayesian nonparametric models. Hierarchical Bayesian nonparametric models are usually built from completely random measures, a Poisson-process-based construction in which the atom weights are independent. Completely random measures imply strong independence assumptions in the corresponding hierarchical model, and these assumptions are often misplaced in real-world settings. Correlated random measures address this limitation. They model correlation within the measure by using a Gaussian process in concert with the Poisson process. With correlated random measures, for example, we can develop a latent feature model for which we can infer both the properties of the latent features and their dependency pattern. We develop several other examples as well. We study a correlated random measure model of pairwise count data. We derive an efficient variational inference algorithm and show improved predictive performance on large datasets of documents, web clicks, and electronic health records. Supplementary materials for this article are available online.
BibTeX
@article{ranganath2016correlated,
  author = {Rajesh Ranganath and David M. Blei},
  title = {{Correlated Random Measures}},
  journal = {Journal of the American Statistical Association},
  year = {2016},
  doi = {10.1080/01621459.2016.1260468},
}

Loading related content...

Figure