Covariance Estimation via Fiducial Inference

W. Jenny Shi, Jan Hannig, Randy C. S. Lai, Thomas C. M. Lee

TL;DR

This paper introduces a generalized fiducial inference (GFI) framework for estimating covariance matrices, offering a powerful alternative to traditional Frequentist and Bayesian methods. By inverting the data-generating process, the authors produce a distribution of "good" estimators that allow for precise uncertainty quantification and the identification of complex sparse structures (cliques) without the need for hand-picked Bayesian priors.

What Problem Does This Paper Address?

Estimating a covariance matrix—the map of how variables in a dataset move together—is a fundamental task in statistics. However, it becomes notoriously difficult in "high-dimensional" settings where the number of variables ($p$) is large relative to the number of observations ($n$).

Current approaches have trade-offs:

  1. Frequentist methods (like graphical LASSO) are great at providing a single "best guess" (point estimate) but struggle to tell you how uncertain that guess is.
  2. Bayesian methods provide a full probability distribution for the estimate (allowing for confidence intervals), but they require a "prior"—an initial assumption about the distribution that can be difficult to justify and may bias the results.

The authors aim to bridge this gap using Fiducial Inference, a concept originally proposed by R.A. Fisher. It seeks to provide the benefits of a Bayesian posterior (a distribution of estimates) without the subjective baggage of a prior.

Methods

The core of the paper is the Generalized Fiducial Distribution (GFD). Instead of using Bayes' Theorem, GFI relies on a "data generating equation": $X = G(U, \theta)$, where $X$ is the data, $\theta$ is the parameter (the covariance matrix), and $U$ is a random component with a known distribution (like standard normal noise).

Key technical contributions include:

  • Jacobian Derivation: The authors derive the complex Jacobian functions required to "invert" the data-generating equation for covariance matrices under different norms ($l_2$ and $l_\infty$).
  • Sparsity Handling: They extend the model to handle clique structures (where variables form independent clusters) and general sparse matrices where many entries are zero.
  • RJMCMC: For cases where the sparsity pattern is unknown, they implement a Reversible Jump Markov Chain Monte Carlo algorithm. This allows the model to "jump" between different dimensions of sparsity to find the most likely structure.
  • Theoretical Proofs: They prove that their fiducial distribution is asymptotically normal and consistent, satisfying the "Fiducial Bernstein-von Mises Theorem."

Datasets

The paper primarily evaluates the method using simulated datasets designed to mimic challenging real-world scenarios:

  • Clique Models: Simulations with $p=200$ variables and $n=1000$ observations, where variables are grouped into 10 or 8 cliques.
  • General Sparse Models: High-dimensional settings (e.g., $p=50, n=50$) with unknown zero locations.
  • Motivation: The authors cite metabolomics and proteomics (e.g., the relationship between gene expression and metabolite levels) as the primary real-world inspiration for these sparse, high-dimensional structures.

Key Insights and Ideas

  • Prior-Free Uncertainty: The GFD acts like a Bayesian posterior but is derived solely from the data-generating mechanism. This is a "killer feature" for researchers who want to report confidence regions for their covariance matrices without defending a specific prior.
  • Structural Discovery: The fiducial approach is remarkably good at identifying "cliques." In simulations, the model was able to recover the true underlying community structure of the variables with high probability.
  • The Jacobian Matters: The choice of norm ($l_2$ vs $l_\infty$) in the Jacobian calculation affects the estimator. The authors show that the $l_2$ norm is particularly effective when combined with a penalty term for model selection.

Key Results

  • Consistency: The authors mathematically prove that as the sample size $n$ grows, the fiducial distribution concentrates around the true covariance matrix.
  • Performance vs. PDSCE: In head-to-head comparisons with the popular Positive Definite Sparse Covariance Estimator (PDSCE), the fiducial estimators were often closer to the truth in terms of the log-determinant and provided better-calibrated coverage.
  • Calibration: Quantile-quantile plots (Figure 3 in the paper) show that the fiducial p-values follow a uniform distribution, indicating that the 95% confidence intervals produced by the method actually contain the truth 95% of the time.

Limitations and Future Work

  • Computational Complexity: The RJMCMC algorithm used for general sparsity is computationally expensive. The authors note that if the matrix is not very sparse, the "computational burden can be significant."
  • Symmetry Assumptions: The current work assumes a squared covariate matrix, which might limit direct application to some types of high-throughput genomic data where the matrix is rectangular.
  • Search Space: To keep the search manageable, the authors had to impose a row/column-wise sparsity upper bound. Developing more sophisticated ways to choose initial states could improve the algorithm's efficiency in the future.

Summary updated

Abstract

As a classical problem, covariance estimation has drawn much attention from the statistical community for decades. Much work has been done under the frequentist and the Bayesian frameworks. Aiming to quantify the uncertainty of the estimators without having to choose a prior, we have developed a fiducial approach to the estimation of covariance matrix. Built upon the Fiducial Berstein-von Mises Theorem (Sonderegger and Hannig 2014), we show that the fiducial distribution of the covariate matrix is consistent under our framework. Consequently, the samples generated from this fiducial distribution are good estimators to the true covariance matrix, which enable us to define a meaningful confidence region for the covariance matrix. Lastly, we also show that the fiducial approach can be a powerful tool for identifying clique structures in covariance matrices.
BibTeX
@inproceedings{shi2024covariance,
  author = {W. Jenny Shi and Jan Hannig and Randy C. S. Lai and Thomas C. M. Lee},
  title = {{Covariance Estimation via Fiducial Inference}},
  booktitle = {UNC},
  year = {2024},
  doi = {10.17615/7zgm-ac90},
}
Figure