Inference for High-dimensional Exponential Family Graphical Models

Jialei Wang, Mladen KolarAISTATS 2016 · OpenReview

Abstract

Probabilistic graphical models have been widely used to model complex systems and aid scientific discoveries. Most existing work on high-dimensional estimation of exponential family graphical models focuses on consistent model selection—identifying the presence or absence of edges. However, performing valid statistical inference, such as constructing confidence intervals for specific edge parameters, remains a significant challenge due to the "irregularity" of high-dimensional estimators. This paper introduces a novel "double selection" procedure that provides $\sqrt{n}$-consistent estimators and valid confidence intervals for individual edges in pairwise exponential family graphical models. The method is shown to be robust to initial model selection errors and achieves performance comparable to an oracle estimator.

Introduction

In modern scientific research, datasets often involve a large number of variables ($p$) relative to the number of observations ($n$). Probabilistic graphical models, such as Gaussian or Ising models, are essential for mapping the relationships within these complex systems. While significant progress has been made in identifying the graph structure (model selection), researchers often require more granular information: the strength of a specific connection and the statistical significance of that connection.

Existing high-dimensional estimators are often biased or highly sensitive to small mistakes in the initial model selection process. This sensitivity makes it difficult to construct reliable confidence intervals. This paper addresses this gap by proposing a robust inference framework for the broad class of pairwise exponential family graphical models, which can handle continuous, binary, and count data.

Methodology: The Double Selection Procedure

The authors propose a three-step Double Selection Procedure designed to estimate a specific edge parameter ($\theta_{ab}$) while treating all other parameters as nuisance variables.

  1. Pilot Estimation: A penalized composite likelihood approach (similar to the Lasso) is used to obtain an initial sparse estimate of the entire graph structure.
  2. Robustification (Neyman Orthogonalization): This critical step involves solving a second optimization problem to find a weighting vector that makes the final estimate insensitive to small errors in the rest of the graph. This "decorrelates" the edge of interest from the noise of other variables.
  3. Refitting: A final restricted maximum likelihood estimation is performed using the union of variables identified in the first two steps.

This framework extends "de-biasing" techniques from linear models to the more complex territory of exponential family graphical models, accommodating non-linear relationships found in binary (Ising) or count (Poisson) data.

Simulation Study and Results

The validity of the procedure was established through extensive synthetic simulations across various graph structures, including chain, nearest neighbor, and Erdős-Rényi graphs. These structures were tested using Gaussian, Ising, and Poisson distributions.

Asymptotic Normality and Coverage

A key theoretical contribution is the proof that the double selection estimator follows a standard Normal distribution as the sample size grows. This asymptotic normality allows for the construction of 95% confidence intervals using standard Z-scores. In simulations with $n=200$ and $p=200$, the method consistently achieved near 95% coverage.

Normal probability plots comparing the Double Selection (DS) estimator and the Oracle estimator for both edge and non-edge parameters.

As shown in Figure 2, the normal probability (Q-Q) plots for the Double Selection (DS) estimator align closely with the diagonal, matching the performance of an "Oracle" estimator that has prior knowledge of the true graph structure. This alignment confirms that the DS estimator is unbiased and follows the predicted normal distribution for both existing edges and non-edges.

Bias Reduction and Efficiency

Compared to "Naive Selection" methods—which perform standard regression after a single Lasso step—the Double Selection method significantly reduces bias. While the resulting confidence intervals are slightly wider than those of an Oracle estimator, they remain highly efficient and practically useful for real-world scientific applications where the true graph is unknown.

Conclusion and Limitations

The Double Selection procedure provides a robust tool for statistical inference in high-dimensional settings. Its primary strength lies in its robustness to model selection mistakes and its generality across the exponential family. However, some limitations remain:

  • Pairwise Assumption: The model currently focuses on interactions between two variables and does not explicitly account for higher-order interactions.
  • Computational Cost: Performing the procedure for every possible edge in a very large graph can be computationally intensive.
  • Sparsity: The theoretical guarantees rely on the assumption that the underlying graph is relatively sparse.

Future work may explore extending this framework to more complex interaction types and improving computational efficiency for massive datasets.

Summary updated

Abstract

Probabilistic graphical models have been widely used to model complex systems and aid scientific discoveries. Most existing work on high-dimensional estimation of exponential family graphical model...
BibTeX
@inproceedings{wang2016inference,
  author = {Jialei Wang and Mladen Kolar},
  title = {{Inference for High-dimensional Exponential Family Graphical Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2016},
  url = {https://openreview.net/forum?id=RTWiWEVxqMd},
}
Figure