Estimation of a covariance matrix with zeros

Subhajyoti Chaudhuri, Mathias Drton, Thomas S. RichardsonBiometrika 2007 · DOI · arXiv

Abstract

We consider estimation of the covariance matrix of a multivariate random vector under the constraint that certain covariances are zero. We first present an algorithm, which we call iterative conditional fitting, for computing the maximum likelihood estimate of the constrained covariance matrix, under the assumption of multivariate normality. In contrast to previous approaches, this algorithm has guaranteed convergence properties. Dropping the assumption of multivariate normality, we show how to estimate the covariance matrix in an empirical likelihood approach. These approaches are then compared via simulation and on an example of gene expression.

Introduction

In multivariate statistics, modeling relationships between variables often involves a covariance matrix. In many applications, domain knowledge suggests that specific pairs of variables are marginally independent, implying that their corresponding entries in the covariance matrix must be zero. These constraints define what are known as covariance graph models.

Example of a covariance graph where edges represent non-zero marginal correlations.

While "concentration graphs" (modeling conditional independence) benefit from well-established fitting algorithms like Iterative Proportional Fitting (IPF), covariance graphs present a greater challenge. They form "curved exponential families" with complex likelihood surfaces. Prior methods, such as Anderson’s algorithm, frequently fail by producing non-positive definite matrices or failing to converge during Maximum Likelihood Estimation (MLE).

Iterative Conditional Fitting (ICF)

The primary contribution of this work is Iterative Conditional Fitting (ICF), a robust algorithm for computing the MLE of a constrained covariance matrix. ICF cycles through each variable, updating its relationship with its "spouses" (correlated variables) while keeping the rest of the matrix fixed.

The algorithm transforms a complex constrained optimization problem into a sequence of simple least-squares regressions using synthetic "pseudo-variables." By leveraging regression steps, ICF ensures that the resulting matrix remains positive definite throughout the process. Theoretically, ICF serves as the "dual" to the IPF algorithm: while IPF fits marginals while fixing conditionals, ICF fits conditionals while fixing marginals.

Empirical Likelihood (EL)

Recognizing that real-world data often deviates from the Gaussian assumption, the authors propose a non-parametric alternative using Empirical Likelihood (EL). This framework does not assume a specific distribution shape. Instead, it assigns weights to each data point and maximizes the likelihood of these weights subject to the zero-covariance constraints. This approach is particularly valuable for data with heavy tails or outliers, common in biological and financial contexts.

Experimental Evaluation

The proposed methods were evaluated using both real-world biological data and extensive simulations.

Yeast Gene Expression

The authors analyzed a dataset involving 134 experiments measuring the expression of 8 genes related to galactose utilization. They compared two models: a sparse model ($G_s$) and a more complex model ($G_d$). The more complex model showed a significantly better fit (deviance of 9.98 vs 33.07), suggesting that some assumed marginal independences in simpler models were likely incorrect.

Simulation Results

Simulations using 1,000 datasets across various sample sizes ($n=10$ to $100$) revealed:

  • Gaussian Settings: The MLE (via ICF) demonstrated the lowest bias and Root Mean Squared Error (RMSE).
  • Non-Gaussian Settings: Using a $t_5$ distribution, the Empirical Likelihood approach outperformed the MLE in terms of RMSE for moderate sample sizes ($n > 20$).

Conclusion

ICF addresses a long-standing stability issue in covariance graph modeling, providing a guaranteed path to valid, positive-definite estimates. While ICF is highly efficient for Gaussian data, the Empirical Likelihood approach offers a robust alternative for non-parametric settings. Future work may extend the logic of ICF to other data types, such as binary graphical models.

Summary updated

Abstract

We consider estimation of the covariance matrix of a multivariate random vector under the constraint that certain covariances are zero. We first present an algorithm, which we call iterative conditional fitting, for computing the maximum likelihood estimate of the constrained covariance matrix, under the assumption of multivariate normality. In contrast to previous approaches, this algorithm has guaranteed convergence properties. Dropping the assumption of multivariate normality, we show how to estimate the covariance matrix in an empirical likelihood approach. These approaches are then compared via simulation and on an example of gene expression.
BibTeX
@inproceedings{chaudhuri2007estimation,
  author = {Subhajyoti Chaudhuri and Mathias Drton and Thomas S. Richardson},
  title = {{Estimation of a covariance matrix with zeros}},
  booktitle = {Biometrika},
  year = {2007},
  doi = {10.1093/biomet/asm007},
}
Figure