Abstract
Conjugate pairs of distributions over infinite dimensional spaces are prominent in statistical learning theory, particularly due to the widespread adoption of Bayesian nonparametric methodologies for a host of models and applications. Much of the existing literature in the learning community focuses on processes possessing some form of computationally tractable conjugacy as is the case for the beta and gamma processes (and, via normalization, the Dirichlet process). For these processes, proofs of conjugacy and requisite derivation of explicit computational formulae for posterior density parameters are idiosyncratic to the stochastic process in question. As such, Bayesian Nonparametric models are currently available for a limited number of conjugate pairs, e.g. the Dirichlet-multinomial and beta-Bernoulli process pairs. In each of these above cases the likelihood process belongs to the class of discrete exponential family distributions. The exclusion of continuous likelihood distributions from the known cases of Bayesian Nonparametric Conjugate models stands as a disparity in the researcher's toolbox. In this paper we first address the problem of obtaining a general construction of prior distributions over infinite dimensional spaces possessing distributional properties amenable to conjugacy. Second, we bridge the divide between the discrete and continuous likelihoods by illustrating a canonical construction for stochastic processes whose Levy measure densities are from positive exponential families, and then demonstrate that these processes in fact form the prior, likelihood, and posterior in a conjugate family. Our canonical construction subsumes known computational formulae for posterior density parameters in the cases where the likelihood is from a discrete distribution belonging to an exponential family.
Introduction
Bayesian Nonparametrics (BNP) has revolutionized statistical learning by allowing models to adapt their complexity to the data. Central to this field is the concept of conjugacy, which ensures that posterior distributions remain within the same family as the prior, facilitating efficient computation. However, while conjugacy is well-established for finite-dimensional models, it has remained fragmented and idiosyncratic in the infinite-dimensional BNP setting. Most existing conjugate BNP models, such as the Beta-Bernoulli or Dirichlet-multinomial pairs, are limited to discrete likelihoods. This paper addresses this gap by providing a general, canonical construction for conjugate BNP models that extends to continuous likelihoods.
The Challenge of Infinite-Dimensional Conjugacy
In standard Bayesian statistics, conjugacy is often a matter of simple arithmetic on parameters. In BNP, however, we work with stochastic processes over infinite-dimensional spaces. Proving conjugacy for processes like the Beta or Gamma process has traditionally required complex, case-specific mathematical derivations. This lack of a unified framework has restricted the development of new conjugate pairs, particularly those involving continuous data.
A Unified "Recipe" via Sufficient Statistics
The authors propose a "Sufficient Statistics Construction" that bridges the gap between finite and infinite-dimensional models. Their approach focuses on Completely Random Measures (CRMs) and their associated Lévy measures. By defining the density of a prior's Lévy measure using the sufficient statistics of an exponential family, they "bake in" conjugacy for likelihoods from that same family.
This construction leverages functional derivatives from variational calculus to show that the moments of these infinite-dimensional processes relate directly to the parameters of their Lévy measures. The result is a set of "canonical triplets"—prior, likelihood, and posterior—where the update rules are as straightforward as their finite-dimensional counterparts. This visualization demonstrates the additive property of the sufficient statistics construction, where the combined process $\nu$ reflects the contributions of its components.
Key Contributions and Results
The theoretical framework established in this paper leads to several significant results:
- Existence and Conjugacy Theorems: The authors prove that for any positive-valued exponential family meeting mild regularity conditions, a corresponding Lévy process prior exists and possesses a conjugate posterior.
- Unified Formulas: They derive general posterior update formulas that mirror the simple addition of sufficient statistics seen in parametric Bayesian models.
- Generalization of Known Cases: The paper shows that the previously idiosyncratic update rules for the Beta and Gamma processes are actually special cases of this general theory.
- New Model Classes: The framework enables the construction of entirely new conjugate BNP models, such as a Pareto process or a Log-normal likelihood paired with a Gamma prior.
Implications and Future Directions
By unlocking continuous likelihoods for conjugate BNP models, this work significantly expands the toolkit available to researchers. While currently restricted to positive-valued exponential families (excluding, for example, the Gaussian distribution), the theory provides a clear path forward. Future work will likely focus on extending this to a broader range of distributions, developing efficient sampling algorithms based on these new constructions, and applying them to complex domains like dynamic graphs and time-series analysis.
Conclusion
This paper demystifies the perceived complexity of BNP conjugacy. By viewing these models through the lens of Lévy measure densities and sufficient statistics, the authors demonstrate that the "magic" of infinite-dimensional updates is fundamentally the same as the arithmetic of standard Bayesian statistics. This unified perspective not only simplifies existing models but also paves the way for a new generation of flexible, computationally efficient Bayesian nonparametric tools.
Summary updated