Construction of Nonparametric Bayesian Models from Parametric Bayes Equations

Peter OrbanzNeurIPS 2009 · OpenReview

This paper by Peter Orbanz provides a rigorous mathematical bridge between the finite-dimensional world of parametric statistics and the infinite-dimensional world of nonparametric Bayesian modeling.

TL;DR

The author introduces a generalization of the classic Kolmogorov extension theorem to conditional probabilities, providing a formal "recipe" for building nonparametric Bayesian models. The paper proves that if your finite-dimensional models are conjugate (specifically within the exponential family), the resulting infinite-dimensional model will also be conjugate and computationally tractable.

What Problem Does This Paper Address?

In machine learning, nonparametric Bayesian models like Gaussian Processes (GP) and Dirichlet Processes (DP) are essential because they allow the "complexity" of a model (the number of parameters) to grow with the data. However, these models are often treated heuristically or constructed case-by-case.

The mathematical "gold standard" for defining stochastic processes is the Kolmogorov extension theorem, which allows you to define an infinite-dimensional process by specifying its finite-dimensional marginals. The problem? Bayesian modeling isn't just about measures; it’s about conditional probabilities (likelihoods and posteriors). The standard theorem doesn't account for the measurability requirements of these conditional arguments. This paper fills that gap, providing a formal framework to ensure that when we "stack" finite-dimensional Bayes equations into an infinite-dimensional model, the result is mathematically sound and statistically useful.

Methods

The author develops two primary theoretical tools:

  1. Extension of Conditional Probabilities (Theorem 2): This is the core contribution. It generalizes Kolmogorov’s theorem to conditional distributions. It states that if a family of finite-dimensional conditional models is "conditionally projective" (meaning they are consistent with each other under projection), there exists a unique infinite-dimensional parametric model.
  2. Functional Conjugacy (Theorem 3): This theorem addresses tractability. It shows that if the finite-dimensional marginals have a tractable "posterior index" (a mapping from data and priors to posterior parameters), this property carries over to the infinite-dimensional limit.

To demonstrate this framework, the author constructs a new nonparametric model for infinite permutations (bijections of the set of natural numbers), extending the Mallows model traditionally used for rank data.

Key Insights and Ideas

  • The "Recipe" for Nonparametrics: To build a new nonparametric model, you don't need to start with complex measure theory. You can start with a system of finite-dimensional parametric Bayes equations. If they are consistent (projective), the infinite-dimensional model is guaranteed to exist.
Examples of nonparametric models constructed as projective limits of finite-dimensional marginals.
  • The Exponential Family Connection: The paper argues that for a nonparametric model to be conjugate (and thus easy to use for inference), it essentially must be constructed from finite-dimensional models in the exponential family.
  • Posterior Indexing: The author introduces the concept of a "posterior index" $T(x, \psi)$, which acts as a generic mapping for updating parameters. This provides a unified way to look at how information from data is incorporated into infinite-dimensional models.
  • Countable Dimensions: The framework specifically targets countable-dimensional spaces. While this sounds restrictive, the author notes that even continuous processes (like GPs on the real line) can often be represented via their values on a countable dense subset (like the rational numbers).

Key Results

  • Rigorous Construction: The paper successfully proves that conditional stochastic processes can be uniquely defined on countable-dimensional spaces.
  • Preservation of Conjugacy: It provides a formal proof that conjugacy is a "projective" property—if your small-scale models are conjugate, your large-scale model is too.
  • New Model Derivation: The author provides the explicit construction for a nonparametric Bayesian analogue of rank-data models, proving the framework can be used to derive novel, complex models for objects like infinite graphs or permutations.

Limitations and Future Work

  • Countable vs. Uncountable: The current results are limited to countable dimensions. Extending this to uncountable dimensions (e.g., directly defining a GP on the entire real line without using a surrogate like the rationals) involves significantly more complex measure theory and is left for future work.
  • Beyond Conjugacy: The paper focuses heavily on conjugacy. The author suggests that future research could explore whether other statistical properties, such as sufficiency or identifiability, carry over from finite to infinite dimensions in a similar way.
  • Product Space Constraints: While powerful, the "product space" approach used here might not be the most intuitive way to describe all models (like Polya trees), even if they can technically be embedded within it.

Summary updated

Abstract

We consider the general problem of constructing nonparametric Bayesian models on infinite-dimensional random objects, such as functions, infinite graphs or infinite permutations. The problem has generated much interest in machine learning, where it is treated heuristically, but has not been studied in full generality in nonparametric Bayesian statistics, which tends to focus on models over probability distributions. Our approach applies a standard tool of stochastic process theory, the construction of stochastic processes from their finite-dimensional marginal distributions. The main contribution of the paper is a generalization of the classic Kolmogorov extension theorem to conditional probabilities. This extension allows a rigorous construction of nonparametric Bayesian models from systems of finite-dimensional, parametric Bayes equations. Using this approach, we show (i) how existence of a conjugate posterior for the nonparametric model can be guaranteed by choosing conjugate finite-dimensional models in the construction, (ii) how the mapping to the posterior parameters of the nonparametric model can be explicitly determined, and (iii) that the construction of conjugate models in essence requires the finite-dimensional models to be in the exponential family. As an application of our constructive framework, we derive a model on infinite permutations, the nonparametric Bayesian analogue of a model recently proposed for the analysis of rank data.
BibTeX
@inproceedings{orbanz2009construction,
  author = {Peter Orbanz},
  title = {{Construction of Nonparametric Bayesian Models from Parametric Bayes Equations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=SJW6Cv-_bB},
}
Figure