Bayesian Inference in Cumulative Distribution Fields

Ricardo SilvaarXiv 2015 · DOI · arXiv

Abstract

One approach for constructing copula functions is by multiplication. Given that products of cumulative distribution functions (CDFs) are also CDFs, an adjustment to this multiplication will result in a copula model. Parameterizing models via products of CDFs has advantages, such as being well-defined for any dimensionality and providing models where small dimensional marginal distributions are easily accessible. We discuss and demonstrate MCMC approaches for estimating such models in a Bayesian context, their application in copula modeling, and how message-passing can be strongly simplified by reframing the problem as a standard graphical model.

Introduction

In multivariate statistics, copulas are essential for modeling dependencies between variables independently of their marginal distributions. A particularly elegant construction method involves multiplying several CDFs. However, performing likelihood-based inference, such as Markov Chain Monte Carlo (MCMC), requires the Probability Density Function (PDF), which is the derivative of the joint CDF. For a product of $p$ functions, the chain rule leads to a sum of $2^p$ terms, creating a combinatorial explosion that renders likelihood calculation intractable for large models unless they possess specific sparse structures.

Previous attempts to address this, such as "Cumulative Distribution Networks" (CDNs), relied on specialized "derivative-sum-product" algorithms. These methods are often difficult to implement and lack integration with modern probabilistic programming frameworks.

Mapping to Standard Graphical Models

The core insight of this work is that the complex derivative of a product of CDFs can be reinterpreted as a marginalization problem over a set of discrete auxiliary variables ($Z$). By introducing indicator variables to track which factor is being differentiated, the problem is transformed from niche calculus into the well-studied domain of Undirected Markov Networks.

Dependency graphs (a, b) and their corresponding factor graphs (c, d) using auxiliary variables Z.

As shown in Figure 2, dependency structures between variables $U$ (graphs a and b) can be mapped to factor graphs (graphs c and d) where the $Z$ variables act as indicators. This "bridge" allows the use of standard message-passing algorithms, such as the Junction Tree algorithm, to compute the likelihood efficiently. The computational complexity is then governed by the "treewidth" of the underlying graph; sparse dependencies (like chains or trees) allow for very fast inference.

Bayesian Estimation Strategies

The paper explores three distinct MCMC approaches for Bayesian estimation:

  • Collapsed Sampler: This method uses dynamic programming to calculate the exact likelihood at each step. It provides high Effective Sample Size (ESS) but is computationally expensive per iteration.
  • Discrete Latent Sampler: This approach jointly samples the model parameters and the discrete $Z$ indicators, offering a balance between computational cost and mixing efficiency.
  • Continuous Latent Sampler: Specifically for Archimedean copulas (e.g., Clayton), this uses a known latent variable representation (e.g., Gamma-distributed variables). While mathematically elegant, it often suffers from poor mixing due to tight coupling between parameters and latent variables.

Experimental Evaluation

The proposed methods were evaluated on two primary tasks:

  • Synthetic Chain Model: A controlled study of a 5-variable chain ($U_1 \leftrightarrow U_2 \leftrightarrow U_3 \leftrightarrow U_4 \leftrightarrow U_5$) generated from Clayton copulas. Results showed that the Collapsed and Discrete Latent samplers converged quickly, while the Continuous Latent sampler failed to converge even after 10,000 iterations.
  • Financial Time Series: Modeling 5 years of daily log-returns for 46 S&P 500 stocks. By partitioning stocks into clusters, the latent variable approach demonstrated that previously impossible "dense" problems could be made computationally feasible.

Conclusion

By reframing Cumulative Distribution Fields as standard graphical models, this research simplifies the implementation of Bayesian inference for complex multivariate dependencies. While challenges remain regarding proposal distributions and high-treewidth models, the use of auxiliary variables and standard message-passing provides a scalable path forward for copula modeling in high dimensions.

Summary updated

Abstract

One approach for constructing copula functions is by multiplication. Given that products of cumulative distribution functions (CDFs) are also CDFs, an adjustment to this multiplication will result in a copula model, as discussed by Liebscher (J Mult Analysis, 2008). Parameterizing models via products of CDFs has some advantages, both from the copula perspective (e.g., it is well-defined for any dimensionality) and from general multivariate analysis (e.g., it provides models where small dimensional marginal distributions can be easily read-off from the parameters). Independently, Huang and Frey (J Mach Learn Res, 2011) showed the connection between certain sparse graphical models and products of CDFs, as well as message-passing (dynamic programming) schemes for computing the likelihood function of such models. Such schemes allows models to be estimated with likelihood-based methods. We discuss and demonstrate MCMC approaches for estimating such models in a Bayesian context, their application in copula modeling, and how message-passing can be strongly simplified. Importantly, our view of message-passing opens up possibilities to scaling up such methods, given that even dynamic programming is not a scalable solution for calculating likelihood functions in many models.
BibTeX
@misc{silva2015bayesian,
  author = {Ricardo Silva},
  title = {{Bayesian Inference in Cumulative Distribution Fields}},
  year = {2015},
  eprint = {1511.02796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.02796},
}
Figure