Graphical models via univariate exponential family distributions

Eunho Yang, Pradeep Ravikumar, Genevera I. Allen, Zhandong LiuJMLR 2015 · OpenReview

TL;DR

This paper introduces a new class of graphical models, "exponential family graphical models," that can naturally handle diverse data types beyond the traditional Gaussian and Ising models. They develop efficient estimation methods and provide theoretical guarantees for learning these models, opening doors for more flexible network inference in various scientific domains.

What Problem Does This Paper Address?

Graphical models, like Markov networks, are powerful tools for representing complex relationships between variables. However, existing popular models, such as Gaussian Markov Random Fields (GMRFs) and Ising models, are often limited to specific data types (real-valued or discrete, respectively). This forces researchers to either transform their data to fit these models or use models that might not be the best fit, potentially leading to inaccurate conclusions. For instance, GMRFs assume thin tails, which can fail to capture extreme events common in financial data. Similarly, count data (like gene expression levels) or skewed continuous data (like time spent on a website) don't naturally fit these standard models. The paper aims to bridge this gap by proposing a general framework that allows for the construction of multivariate graphical models directly from a wide range of univariate exponential family distributions, such as Poisson, exponential, and gamma.

Methods

The core innovation lies in constructing multivariate graphical models by specifying the node-wise conditional distributions to follow univariate exponential families. This approach, grounded in the Hammersley-Clifford theorem, ensures that the resulting joint distribution factors according to the graph structure. The authors then propose a class of M-estimators for learning these models, specifically employing an $l_1$-regularized conditional log-likelihood maximization (a form of penalized regression) to encourage sparsity and recover the graph structure. A significant contribution is the rigorous statistical analysis providing sparsistency guarantees for these M-estimators, meaning they can recover the true graph structure with high probability, even in high-dimensional settings where the number of variables ($p$) is much larger than the number of samples ($n$). This analysis extends previous work on Gaussian and Ising models to a more general class of models.

Datasets

The paper uses simulated data for experiments and evaluation. Specifically, they generate data from:

  • Exponential graphical models
  • Poisson graphical models
  • Gaussian graphical models

They also present two real-world data examples:

  • A meta-miRNA inhibitory network for breast cancer, estimated using a Poisson graphical model.
  • A cell signaling network, estimated using an exponential graphical model.

Key Insights and Ideas

  • Flexibility through Univariate Distributions: The key insight is that by leveraging the rich family of univariate exponential distributions, we can construct a much broader and more flexible class of multivariate graphical models. This directly addresses the limitations of existing models for non-Gaussian and non-categorical data.
  • Node-wise Estimation for Global Structure: The paper demonstrates that learning the global graph structure can be effectively achieved by focusing on estimating local node-neighborhoods using penalized conditional likelihoods. This is a computationally tractable approach.
  • Theoretical Guarantees for General Models: Providing rigorous statistical guarantees (sparsistency) for learning these new graphical models is crucial. This work extends existing theoretical results to a more general setting, increasing confidence in the applicability of these models.
  • Poisson and Exponential Models Capture Negative Dependencies: A notable finding is that the Poisson and exponential graphical models, due to their parameter constraints, are particularly adept at capturing negative conditional dependencies between variables. This is highlighted in their experimental results.

Key Results

  • New Model Class: The paper introduces and formalizes "exponential family graphical models," a general class that encompasses and extends existing models like Gaussian MRFs and Ising models.
  • Sparsistency Guarantees: Theoretical results demonstrate that the proposed M-estimators can recover the true graph structure with high probability, even in high-dimensional settings.
  • Empirical Performance: Simulation studies show that the proposed Poisson and exponential M-estimators outperform Gaussian M-estimators when data is generated from Poisson or exponential distributions, especially in high-dimensional regimes ($n < p$).
  • Real-World Applications: The models are successfully applied to real biological data, revealing insights into miRNA inhibitory networks and cell signaling pathways. For instance, the Poisson model identified key inhibitory relationships in breast cancer miRNAs, and the exponential model highlighted PKA as a major protein inhibitor in cell signaling.

Limitations and Future Work

  • Goodness-of-Fit Tests: The authors acknowledge the lack of rigorous statistical tests for assessing the goodness-of-fit of these general exponential family graphical models when the network structure is unknown. They propose a heuristic but suggest that developing an exact test is an important future direction.
  • Parameter Space Restrictions: For certain exponential family distributions (like Poisson and exponential), the models can only capture negative conditional dependencies due to parameter constraints. This limitation might be undesirable in practice if positive dependencies are also present. Future work could explore relaxing these restrictions for specific distributions.
  • Multi-parameter Exponential Families: The current work focuses on single-parameter exponential families. Investigating the implications of using multi-parameter exponential families for graphical models is suggested as a future research avenue.
  • Exact Test for Model Fit: The paper mentions the need for more rigorous likelihood ratio tests for model fit, especially when estimating neighborhoods.

Summary updated

Abstract

Undirected graphical models, or Markov networks, are a popular class of statistical models, used in a wide variety of applications. Popular instances of this class include Gaussian graphical models and Ising models. In many settings, however, it might not be clear which subclass of graphical models to use, particularly for non-Gaussian and non-categorical data. In this paper, we consider a general sub-class of graphical models where the node-wise conditional distributions arise from exponential families. This allows us to derive multivariate graphical model distributions from univariate exponential family distributions, such as the Poisson, negative binomial, and exponential distributions. Our key contributions include a class of M-estimators to fit these graphical model distributions; and rigorous statistical analysis showing that these M-estimators recover the true graphical model structure exactly, with high probability. We provide examples of genomic and proteomic networks learned via instances of our class of graphical models derived from Poisson and exponential distributions.
BibTeX
@article{yang2015graphical,
  author = {Eunho Yang and Pradeep Ravikumar and Genevera I. Allen and Zhandong Liu},
  title = {{Graphical models via univariate exponential family distributions}},
  journal = {J. Mach. Learn. Res.},
  year = {2015},
  url = {https://openreview.net/forum?id=JVY8Mxvw74s},
}
Figure