Simultaneous Inference for Pairwise Graphical Models with Generalized Score Matching

Ming Yu, Varun Gupta, Mladen KolarJMLR 2020 · OpenReview

TL;DR

This paper introduces a new statistical method for understanding complex relationships (graphical models) in data, even when the underlying mathematical structure is hard to compute. The method allows for reliable inference and hypothesis testing on these relationships, particularly in high-dimensional settings, and is robust to model selection errors.

What Problem Does This Paper Address?

Probabilistic graphical models are powerful tools for representing dependencies between variables in a system, finding applications in fields like biology and finance. While significant progress has been made in estimating the parameters of these models and selecting the correct structure, a crucial gap exists in quantifying the uncertainty associated with these estimates and selections. This uncertainty quantification is vital for making reliable statistical inferences, such as constructing confidence intervals or performing hypothesis tests.

A major hurdle in this area is the intractability of computing the "normalizing constant" (or partition function) for many graphical models, especially those with complex dependencies or non-standard distributions. Existing methods for statistical inference often rely on models where this constant is easily calculable (like Gaussian graphical models). This paper aims to bridge this gap by developing a method that can perform statistical inference for a broader class of graphical models, including those with intractable normalizing constants, using a technique called generalized score matching.

Methods

The core of this paper's contribution lies in developing a novel estimator for edge parameters in pairwise graphical models. The approach is built upon generalized Hyvärinen scoring rule, a technique that is particularly useful when the normalizing constant is difficult to compute. The proposed method involves a three-step procedure:

  1. Pilot Estimator: A preliminary estimate of the edge parameter is obtained by minimizing a penalized score matching loss function. This step uses regularization (like L1-norm penalty) to handle high-dimensional settings.
  2. Debiasing/Refitting: A second step refines the estimate by solving a constrained optimization problem. This is crucial for creating a "regular" estimator that is asymptotically normal, addressing issues that arise from model selection in the first step.
  3. Final Estimator: The final estimator is obtained by solving another optimization problem, incorporating information from the previous steps.

A key innovation is the development of a Bahadur-type representation for the final estimator. This representation allows the authors to prove that their estimator is $\sqrt{n}$-consistent (meaning its error decreases proportionally to the square root of the sample size) and asymptotically normal. This asymptotic normality is the foundation for constructing confidence intervals and performing hypothesis tests.

Furthermore, the paper extends these methods to simultaneous inference, enabling hypothesis tests on multiple edge parameters at once. This is achieved using a Gaussian multiplier bootstrap procedure, which is robust to model selection mistakes and can handle high-dimensional settings. The method is also adapted for non-negative data using a generalized l-score matching rule.

Datasets

The paper evaluates its proposed methods using:

  • Synthetic datasets: Generated from four different exponential family distributions (Gaussian, Truncated Gaussian, Exponential, and Normal conditionals) to demonstrate the performance of the inference procedures under controlled conditions.
  • Protein Signaling Dataset: A real-world dataset containing protein presence data from flow cytometry experiments (p=11 proteins, n=7466 cells). This dataset is used to illustrate the application of the method to infer network structures and dependencies in biological systems.

Key Insights and Ideas

  • Bridging the Intractability Gap: The most significant insight is the successful application of statistical inference techniques (confidence intervals, hypothesis testing) to graphical models where the normalizing constant is intractable. This significantly expands the applicability of graphical model analysis.
  • Robustness to Model Selection: The proposed three-step estimation procedure and the use of a decorrelated score function make the inference robust to errors in model selection, a common challenge in high-dimensional settings.
  • Simultaneous Inference Power: The development of a simultaneous inference procedure allows for testing complex hypotheses involving multiple parameters, such as determining if a node is isolated or if graph structures differ between datasets.
  • Generalized Score Matching Versatility: The use of generalized score matching, particularly with the l(x) = log(x+1) function for non-negative data, improves estimation efficiency and practical performance compared to standard score matching.

Key Results

  • The proposed estimator for edge parameters is proven to be $\sqrt{n}$-consistent and asymptotically normal under mild conditions.
  • The method allows for the construction of confidence intervals and the performance of hypothesis tests for individual edge parameters.
  • A procedure for simultaneous inference is developed, enabling hypothesis tests on multiple edges connected to a specific node. This procedure is shown to be effective in applications like testing for isolated nodes, support recovery, and testing for differences between graphical models.
  • Extensive simulation studies on synthetic data demonstrate that the proposed methods achieve nominal coverage rates for confidence intervals and controlled Type I errors for hypothesis tests across various model types and data sizes.
  • The application to the Protein Signaling Dataset successfully infers network structure and identifies significant dependencies, consistent with prior research.

Limitations and Future Work

  • Sample Complexity: The paper notes that the sample complexity for simultaneous inference can be large, especially in high-dimensional settings. While the theoretical results hold, practical performance might require substantial sample sizes.
  • General L Extension: While the paper extends the methodology to general L (number of sufficient statistics per edge), the complexity of the procedure increases.
  • Discrete Data: The current method is developed for continuous data. Extending the inferential techniques to discrete-valued data is identified as an interesting future direction.
  • Other Model Types: The authors suggest exploring inferential techniques for other advanced graphical model structures, such as multi-attribute graphical models, time-varying graphical models, and graphical models with latent variables or confounders.
  • Constrained Inference: Incorporating constraints into the model and performing constrained inference is also mentioned as a future research avenue.

Summary updated

Abstract

Probabilistic graphical models provide a flexible yet parsimonious framework for modeling dependencies among nodes in networks. There is a vast literature on parameter estimation and consistent model selection for graphical models. However, in many of the applications, scientists are also interested in quantifying the uncertainty associated with the estimated parameters and selected models, which current literature has not addressed thoroughly. In this paper, we propose a novel estimator for statistical inference on edge parameters in pairwise graphical models based on generalized Hyvarinen scoring rule. Hyvarinen scoring rule is especially useful in cases where the normalizing constant cannot be obtained efficiently in a closed form, which is a common problem for graphical models, including Ising models and truncated Gaussian graphical models. Our estimator allows us to perform statistical inference for general graphical models whereas the existing works mostly focus on statistical inference for Gaussian graphical models where finding normalizing constant is computationally tractable. Under mild conditions that are typically assumed in the literature for consistent estimation, we prove that our proposed estimator is $\sqrt{n}$-consistent and asymptotically normal, which allows us to construct confidence intervals and build hypothesis tests for edge parameters. Moreover, we show how our proposed method can be applied to test hypotheses that involve a large number of model parameters simultaneously. We illustrate validity of our estimator through extensive simulation studies on a diverse collection of data-generating processes.
BibTeX
@article{yu2020simultaneous,
  author = {Ming Yu and Varun Gupta and Mladen Kolar},
  title = {{Simultaneous Inference for Pairwise Graphical Models with Generalized Score Matching}},
  journal = {J. Mach. Learn. Res.},
  year = {2020},
  url = {https://openreview.net/forum?id=NDk6YLZYI3},
}
Figure