Maximum A Posteriori Covariance Estimation Using a Power Inverse Wishart Prior

Søren Feodor Nielsen, Jon SporringarXiv 2012 · DOI · arXiv

Abstract

The estimation of the covariance matrix is an initial step in many multivariate statistical methods such as principal components analysis and factor analysis, but in many practical applications the dimensionality of the sample space is large compared to the number of samples, and the usual maximum likelihood estimate is poor. Typically, improvements are obtained by modelling or regularization. From a practical point of view, these methods are often computationally heavy and rely on approximations. As a fast substitute, we propose an easily calculable maximum a posteriori (MAP) estimator based on a new class of prior distributions generalizing the inverse Wishart prior, discuss its properties, and demonstrate the estimator on simulated and real data.

1. Introduction

In multivariate statistics, the covariance matrix is essential for understanding the relationships between variables. However, researchers frequently encounter the "small-$n$, large-$p$" problem, where the number of variables ($p$) is large relative to the number of samples ($n$). In these high-dimensional settings, the standard Maximum Likelihood Estimate (MLE)—the sample covariance matrix—becomes highly biased. It tends to "stretch" the spectrum, overestimating large eigenvalues and underestimating small ones, often leading to singular matrices that are impossible to invert.

2. The Power Inverse Wishart (PIW) Prior

To address the limitations of traditional regularization, this paper introduces the Power Inverse Wishart (PIW) distribution. This new class of priors generalizes the standard Inverse Wishart (IW) distribution by introducing a power parameter $q$.

When $q=1$, the distribution recovers the standard IW prior. However, for $q > 1$, the regularization becomes non-linear and convex. This allows the estimator to treat small and large eigenvalues differently, applying more aggressive shrinkage at the "tails" of the spectrum where MLE bias is most destructive. The PIW-MAP estimator effectively imposes a "floor" to ensure invertibility while pulling exaggerated eigenvalues back toward reality.

3. MAP Estimation and Computational Efficiency

A core contribution of this work is the derivation of a computationally efficient Maximum A Posteriori (MAP) estimator. Unlike many Bayesian methods that require slow Markov Chain Monte Carlo (MCMC) simulations, the PIW-MAP can be calculated quickly. For the specific case of $q=2$, the authors provide a closed-form solution, making it a practical "drop-in" replacement for the sample covariance matrix that maintains robustness without a heavy computational "Bayesian tax."

4. Key Results and Applications

The estimator was evaluated using both simulated data and real-world shape data:

  • Simulated Data: PIW-MAP consistently showed lower quadratic risk (estimation error) compared to the standard MLE and outperformed the standard IW-MAP when the eigenvalue floor was set appropriately.
  • Real-World Shape Data: The method was applied to 96-dimensional vectors representing the shape of human femur notches ($n=21$). While the standard MLE was singular and unusable for prediction, the PIW-MAP provided a regularized matrix that allowed for the successful prediction of missing bone segments.

5. Conclusion and Future Work

The Power Inverse Wishart prior offers a robust framework for high-dimensional covariance estimation. While the current work focuses on integer values for $q$, future research may explore fractional powers for even finer regularization control. Additionally, automated methods for hyperparameter tuning ($\alpha, m, q$) remain an area for further development to balance the influence of the prior and the observed data.

Summary updated

Abstract

The estimation of the covariance matrix is an initial step in many multivariate statistical methods such as principal components analysis and factor analysis, but in many practical applications the dimensionality of the sample space is large compared to the number of samples, and the usual maximum likelihood estimate is poor. Typically, improvements are obtained by modelling or regularization. From a practical point of view, these methods are often computationally heavy and rely on approximations. As a fast substitute, we propose an easily calculable maximum a posteriori (MAP) estimator based on a new class of prior distributions generalizing the inverse Wishart prior, discuss its properties, and demonstrate the estimator on simulated and real data.
BibTeX
@misc{nielsen2012maximum,
  author = {Søren Feodor Nielsen and Jon Sporring},
  title = {{Maximum A Posteriori Covariance Estimation Using a Power Inverse Wishart Prior}},
  year = {2012},
  eprint = {1206.2054},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1206.2054},
}
Figure