Mixed Cumulative Distribution Networks

Ricardo Bezerra de Andrade e Silva, Charles Blundell, Yee Whye Teh

Introduction: The Marginalization Problem

In probabilistic graphical modeling, Directed Acyclic Graphs (DAGs) are the standard for representing causal relationships. However, DAGs suffer from a significant limitation: they are not "closed" under marginalization. When certain variables in a DAG are hidden or unobserved, the independence structure among the remaining variables often cannot be captured by another DAG.

To address this, researchers utilize Acyclic Directed Mixed Graphs (ADMGs). These graphs incorporate directed edges ($\to$) for direct effects and bi-directed edges ($\leftrightarrow$) to represent latent common causes. While mathematically robust, ADMGs have historically been difficult to parameterize, particularly for non-Gaussian or mixed data types.

Marginalization of a causal DAG (a) into an ADMG (b) with bi-directed edges representing latent variables.

Methodology: Mixed Cumulative Distribution Networks

The authors propose Mixed Cumulative Distribution Networks (MCDNs) as a flexible solution for parameterizing ADMGs. The framework is built upon three technical pillars:

  1. District Factorization: The joint distribution of an ADMG is decomposed into "districts"—maximal sets of nodes connected by bi-directed edges. This allows the global problem to be treated as a product of local conditional distributions.
  2. Reduction to Barren Districts: The authors employ a transformation using artificial vertices to simplify any ADMG into one where no node in a district is a parent of another in the same district, making the underlying mathematics tractable.
  3. Copula-based Parameterization: Within each district, copulas are used to define the dependence structure. This approach allows researchers to specify marginal distributions (e.g., Poisson, Gaussian, or Bernoulli) independently of the interaction structure, ensuring that marginal independence constraints are naturally satisfied.

Learning is conducted via a hybrid Bayesian approach, where univariate marginals are fitted first, followed by Markov chain Monte Carlo (MCMC) estimation of the copula parameters.

Experimental Evaluation

The MCDN framework was tested against traditional Gaussian and probit-based ADMG models using seven datasets from the UCI Machine Learning Repository, covering binary, ordinal, and continuous data types.

Comparison of average log-predictive probabilities across various datasets.

As shown in the results table, the Copula MCDN consistently outperformed the baseline models. Notable improvements were observed in the Ionosphere and Parkinsons datasets, where the flexibility of the copula-based approach allowed for a much better fit than the restrictive additive error structures of Gaussian models.

Conclusion and Future Work

MCDNs represent a significant advancement in the parameterization of latent variable models. By shifting the focus from factorizing probability density functions (PDFs) to cumulative distribution functions (CDFs), the authors provide a natural way to handle the marginal independence constraints of ADMGs.

Future research will focus on scaling the model for denser graphs—addressing the exponential complexity of the inclusion-exclusion formula for discrete data—and integrating MCDNs with automated structure learning algorithms to discover causal relationships directly from data.

Summary updated

Abstract

Directed acyclic graphs (DAGs) are a popular framework to express multivariate probability distributions. Acyclic directed mixed graphs (ADMGs) are generalizations of DAGs that can succinctly capture much richer sets of conditional independencies, and are especially useful in modeling the effects of latent variables implicitly. Unfortunately there are currently no good parameterizations of general ADMGs. In this paper, we apply recent work on cumulative distribution networks and copulas to propose one one general construction for ADMG models. We consider a simple parameter estimation approach, and report some encouraging experimental results.
BibTeX
@inproceedings{silva2011mixed,
  author = {Ricardo Bezerra de Andrade e Silva and Charles Blundell and Yee Whye Teh},
  title = {{Mixed Cumulative Distribution Networks}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2011},
  url = {https://openreview.net/forum?id=AdzNh0nVDN6},
}
Figure