Random Function Priors for Correlation Modeling

Aonan Zhang, John W. Paisley

Abstract

The likelihood model of high-dimensional data $X_n$ can often be expressed as $p(X_n|Z_n,\theta)$, where $\theta := (\theta_k)_{k \in [K]}$ is a collection of hidden features shared across objects, and $Z_n$ is a non-negative factor loading vector where $Z_{nk}$ indicates the strength of feature $\theta_k$ for object $n$. We introduce random function priors for $Z_n$ to model correlations among its $K$ dimensions, a method we call population random measure embedding (PRME). Our model can be viewed as a generalized paintbox model using random functions and can be learned efficiently with neural networks via amortized variational inference. We derive this Bayesian nonparametric method by applying a representation theorem on separately exchangeable discrete random measures.

Introduction

In high-dimensional data analysis, such as topic modeling, objects are typically represented using a set of latent features. A major challenge in this field is modeling the correlation between these features. Traditional approaches often face a trade-off: parametric models (like Correlated Topic Models) use fixed structures that may be too rigid for complex relationships, while standard nonparametric models (like the Indian Buffet Process) often assume feature independence to maintain mathematical tractability.

PRME addresses this by providing a flexible, nonparametric prior that can capture arbitrarily complex correlations while remaining computationally efficient. By treating feature loadings as outcomes of random functions, the framework bridges the gap between flexible Bayesian priors and the scalable inference typical of Variational Autoencoders (VAEs).

Methodology

The core innovation of PRME is the embedding of latent loadings into a separately exchangeable discrete random measure. This approach is supported by a representation theorem showing that these measures can be decomposed into a coupling of a Poisson process and random functions.

Decomposition of the random measure $\xi$ into components representing the nonparametric feature indexing and the correlation-modeling random functions.

Key Technical Components

  • Representation Theorem: The random measure is represented as a functional form coupling a Poisson process (handling the infinite feature space) with random functions (modeling the correlations).
  • Neural Network Priors: Deep neural networks parameterize these random functions. An inference network (encoder) maps data to a latent space, while a decoder network determines how features correlate based on that representation.
  • Amortized Variational Inference: By leveraging stochastic gradient methods, PRME scales to datasets with millions of documents, a significant improvement over traditional complex Bayesian nonparametric models.
  • Generalized Paintboxes: The model generalizes the "paintbox" analogy from probability theory, where features are "painted" onto a unit square using non-linear functions learned directly from the data.

Experimental Evaluation

The authors evaluated PRME on standard text corpora, including the New York Times (NYT) (with a full 1M document set for scalability tests), 20Newsgroups, and NeurIPS conference papers.

Key Results

  • Predictive Performance: PRME consistently achieved lower perplexity compared to strong baselines such as the Hierarchical Dirichlet Process (HDP) and the Discrete Infinite Logistic Normal (DILN) distribution.
  • Representational Power: Experiments demonstrated that increasing the depth of the neural networks (up to 8 layers) generally improved performance, confirming that the model effectively leverages deep learning's capacity.
  • Scalability: Using an online stochastic inference algorithm, the model successfully processed the 1-million-document NYT dataset, converging significantly faster than traditional batch methods.

Discussion and Insights

A key theoretical insight of the paper is the "Empty Model" trap (Proposition 1): assuming a binary matrix is both row/column exchangeable with finite row sums leads to a trivial model of all zeros. This justifies the necessity of the more complex random measure framework.

The Topic Paintbox visualization demonstrates how the model perceives correlations. Overlapping regions represent topics that the model has learned are likely to co-occur.

The "Topic Paintboxes" provide a visual representation of learned feature relationships. Overlapping regions in these 2D visualizations indicate topics that the model has identified as highly correlated, such as "Music," "Concert," and "Orchestra," based on their co-occurrence patterns in the data.

Conclusion

PRME introduces a powerful framework for modeling correlations in latent feature spaces by combining Bayesian nonparametrics with deep learning. While the current reliance on a non-differentiable Poisson process presents a minor limitation for end-to-end training, future work aims to address this and extend the framework to other domains like relational data and network modeling.

Summary updated

Abstract

The likelihood model of high dimensional data $X_n$ can often be expressed as $p(X_n|Z_n,\theta)$, where $\theta\mathrel{\mathop:}=(\theta_k)_{k\in[K]}$ is a collection of hidden features shared across objects, indexed by $n$, and $Z_n$ is a non-negative factor loading vector with $K$ entries where $Z_{nk}$ indicates the strength of $\theta_k$ used to express $X_n$. In this paper, we introduce random function priors for $Z_n$ for modeling correlations among its $K$ dimensions $Z_{n1}$ through $Z_{nK}$, which we call \textit{population random measure embedding} (PRME). Our model can be viewed as a generalized paintbox model~\cite{Broderick13} using random functions, and can be learned efficiently with neural networks via amortized variational inference. We derive our Bayesian nonparametric method by applying a representation theorem on separately exchangeable discrete random measures.
BibTeX
@inproceedings{zhang2019random,
  author = {Aonan Zhang and John W. Paisley},
  title = {{Random Function Priors for Correlation Modeling}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=rJZsZnZ_-H},
}

Loading related content...

Figure