A conjugate prior for the Dirichlet distribution

Jean-Marc AndreoliarXiv 2018 · OpenReview

Abstract

This note investigates a conjugate class for the Dirichlet distribution class in the exponential family. We formally define the "Boojum" distribution, establish its convergence properties, and propose a numerical framework for its computation.

Introduction

In Bayesian statistics, the Dirichlet distribution is the standard conjugate prior for multinomial distributions. However, performing Bayesian inference on the parameters of the Dirichlet distribution itself is mathematically challenging. While the theory of the exponential family guarantees the existence of a conjugate prior for any member, the specific prior for the Dirichlet distribution has remained largely theoretical due to its complexity. This paper formally defines this distribution, dubbed the "Boojum" distribution, and addresses the practical hurdles of its use.

The Boojum Distribution

The Boojum distribution, $Boojum(m, r)$, is derived systematically within the exponential family framework. It is characterized by a shape parameter $m$ and a rate vector $r$. Its density is proportional to the multivariate Beta function raised to the power of $-m$, directly linking it to the normalizing constant of the Dirichlet distribution.

A key property of the Boojum distribution is its conjugacy: if a Boojum prior is used with Dirichlet-distributed data, the posterior remains within the Boojum family. Specifically, given a prior $(m, r)$ and $N$ Dirichlet samples $y_{1:N}$, the posterior parameters are updated as $(m + N, r - \sum \log y_n)$.

Convergence and Properness

A significant contribution of this work is the rigorous analysis of the conditions under which the Boojum distribution is "proper"—meaning its normalizing constant $Z(m, r)$ is finite. Unlike many common distributions, the Boojum is not valid for all positive parameters.

Parameter space regions where the Boojum distribution is proper.

As illustrated in Figure 1, the distribution is proper if $r_k > 0$ and $m > -1$. However, for $m > 0$, a non-trivial inequality constraint must be met: $\sum \exp(-r_k/m) < 1$. This boundary defines the valid parameter space for Bayesian inference.

Numerical Computation on the Simplex

Because the normalizing constant $Z(m, r)$ lacks a closed-form solution, numerical approximation is required. The paper proposes a method to tile the $(K-1)$-dimensional simplex with Voronoi cells to facilitate computation.

Tiling the simplex with Voronoi cells for FFT-based integration.

By leveraging this geometric structure, Fast Fourier Transforms (FFTs) can be used to compute the necessary convolutions efficiently. For integration over the shape parameter, importance sampling using a Gamma distribution as a pivot is suggested to handle the computational load.

Conclusion

The formalization of the Boojum distribution provides a rigorous foundation for hierarchical Bayesian models involving Dirichlets, such as Latent Dirichlet Allocation (LDA). While computational complexity remains a challenge for high-dimensional data, the proposed FFT-based numerical framework offers a path forward for practitioners to move beyond point estimates of Dirichlet parameters toward fully Bayesian learning.

Summary updated

Abstract

This note investigates a conjugate class for the Dirichlet distribution class in the exponential family.
BibTeX
@misc{andreoli2018conjugate,
  author = {Jean-Marc Andreoli},
  title = {{A conjugate prior for the Dirichlet distribution}},
  year = {2018},
  eprint = {1811.05266},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.05266},
}
Figure