M-Estimators of Scatter with Eigenvalue Shrinkage

Esa Ollila, Daniel P. Palomar, Frédéric PascalarXiv 2020 · DOI · arXiv

Abstract

A popular regularized (shrinkage) covariance estimator is the shrinkage sample covariance matrix (SCM) which shares the same set of eigenvectors as the SCM but shrinks its eigenvalues toward its grand mean. In this paper, a more general approach is considered in which the SCM is replaced by an M-estimator of scatter matrix and a fully automatic data adaptive method to compute the optimal shrinkage parameter with minimum mean squared error is proposed. Our approach permits the use of any weight function such as Gaussian, Huber's, or t weight functions, all of which are commonly used in M-estimation framework. Our simulation examples illustrate that shrinkage M-estimators based on the proposed optimal tuning combined with robust weight function do not loose in performance to shrinkage SCM estimator when the data is Gaussian, but provide significantly improved performance when the data is sampled from a heavy-tailed distribution.

Introduction

In multivariate statistics, estimating the covariance matrix is a fundamental task. The standard tool is the Sample Covariance Matrix (SCM). However, the SCM has two major "Achilles' heels": high dimensionality and a lack of robustness. When the number of variables ($p$) is large relative to the number of observations ($n$), the SCM's eigenvalues become biased. Furthermore, the SCM is extremely sensitive to outliers and heavy-tailed distributions. While "shrinkage" estimators, such as the Ledoit-Wolf estimator, address the dimensionality problem by pulling eigenvalues toward their mean, they are typically based on the SCM and remain fragile against non-Gaussian data. Conversely, robust M-estimators handle heavy tails well but do not traditionally incorporate optimal eigenvalue shrinkage. This paper bridges that gap by proposing a Shrinkage M-estimator of Scatter.

Methodology

The core technical contribution of this work is the derivation of an optimal shrinkage parameter ($\beta$) for robust M-estimators. Instead of shrinking the SCM, shrinkage is applied to a robust M-estimator of scatter.

Because M-estimators are defined implicitly, finding the exact Mean Squared Error (MSE) is difficult. The authors utilize a "1-step estimator" as a mathematical proxy to derive an approximate "oracle" shrinkage parameter. This formula for the optimal $\beta$ is universal and works for any weight function $u(t)$. Specifically, the paper highlights:

  • Huber’s weights: A hybrid that treats "central" data normally and "dampens" outliers.
  • t-MLE weights: Based on the Student’s $t$-distribution, which adaptively handles heavy tails.

The proposed method is fully automatic and data-adaptive, estimating all necessary constants directly from the data.

Experimental Results

The proposed estimators were evaluated using synthetic simulations with an AR(1) covariance structure.

Performance in Gaussian Environments

One significant finding is that robust shrinkage estimators do not lose performance compared to standard methods when the data is Gaussian. As shown in the figure below, the proposed robust shrinkage estimators (Huber and $t$-MLE) match the performance of the gold-standard Ledoit-Wolf (LW) estimator.

NMSE comparison under Gaussian distribution, showing that robust estimators perform as well as the standard SCM-based shrinkage.

Robustness in Heavy-Tailed Environments

The advantages of the robust framework are most apparent in heavy-tailed environments, such as those modeled by $t$-distributions ($t_5$ and $t_3$). In these cases, standard SCM-based shrinkage (Gauss/LW) fails significantly. In contrast, the robust shrinkage M-estimators maintain low Normalized Mean Squared Error (NMSE).

NMSE comparison under heavy-tailed t-distributions, highlighting the superior performance of robust shrinkage M-estimators.

The $t$-MLE shrinkage estimator consistently outperformed the Huber-based version because it better adapted to the specific tail-behavior of the data by learning the degrees of freedom.

Conclusion

This paper introduces a robust framework that combines M-estimators of scatter with eigenvalue shrinkage. The results demonstrate that robust estimators with optimal shrinkage perform identically to the SCM on Gaussian data while being vastly superior on heavy-tailed data. Future work will focus on extending this framework to the $p > n$ case, where the number of variables exceeds the number of observations.

Summary updated

Abstract

A popular regularized (shrinkage) covariance estimator is the shrinkage sample covariance matrix (SCM) which shares the same set of eigenvectors as the SCM but shrinks its eigenvalues toward its grand mean. In this paper, a more general approach is considered in which the SCM is replaced by an M-estimator of scatter matrix and a fully automatic data adaptive method to compute the optimal shrinkage parameter with minimum mean squared error is proposed. Our approach permits the use of any weight function such as Gaussian, Huber's, or t weight functions, all of which are commonly used in M-estimation framework. Our simulation examples illustrate that shrinkage M-estimators based on the proposed optimal tuning combined with robust weight function do not loose in performance to shrinkage SCM estimator when the data is Gaussian, but provide significantly improved performance when the data is sampled from a heavy-tailed distribution.
BibTeX
@misc{ollila2020m,
  author = {Esa Ollila and Daniel P. Palomar and Frédéric Pascal},
  title = {{M-Estimators of Scatter with Eigenvalue Shrinkage}},
  year = {2020},
  eprint = {2002.04996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.04996},
}
Figure