Spectral Shrinkage of Tyler's $M$-Estimator of Covariance Matrix

Arnaud Breloy, Esa Ollila, Frédéric Pascal2019 · DOI

TL;DR

This paper introduces a novel method for estimating covariance matrices, particularly useful when dealing with high-dimensional data and known spectral structures. By applying a "spectral shrinkage" penalty to Tyler's M-estimator, they ensure that the estimated eigenvalues are pulled towards a desired target spectrum without distorting the eigenvectors. This leads to more accurate and robust covariance estimates, especially in challenging scenarios.

What Problem Does This Paper Address?

Estimating covariance matrices is fundamental in many statistical and machine learning tasks. Traditional methods like the sample covariance matrix can be unreliable when the number of variables ($p$) is large compared to the number of samples ($n$), a common issue in modern datasets. Robust estimators like Tyler's M-estimator offer better performance in the presence of outliers, but they also face challenges with "small $n$, large $p$" problems and cannot be computed when $n < p$.

Shrinkage methods have been proposed to address this by regularizing the estimator, pulling it towards a target matrix. However, existing shrinkage methods often shrink both the eigenvalues and eigenvectors, which can be undesirable if the goal is to preserve the underlying eigenvector structure while adjusting the eigenvalue distribution. This paper tackles the problem of how to shrink the spectrum (eigenvalues) of a robust covariance estimator towards a specific target without affecting the eigenvectors, thereby better exploiting known spectral structures like low-rank properties.

Methods

The core contribution is a new regularized version of Tyler's M-estimator, termed the Spectral Shrinkage of Tyler's M-estimator (SRTy). The key innovation lies in the penalty term used for regularization. Instead of a penalty that affects the entire matrix, they introduce a unitary invariant penalty that only operates on the eigenvalues. This penalty shrinks the estimated eigenvalues towards a predefined target spectrum, denoted by $A_T$.

The objective function for the SRTy estimator is formulated as the standard Tyler's M-estimator cost function plus this new spectral shrinkage penalty. The authors derive a closed-form fixed-point equation that characterizes the solution. To compute this estimator efficiently, they develop a Majorization-Minization (MM) algorithm. This algorithm iteratively refines the estimate by minimizing a surrogate function, guaranteeing monotonic convergence of the objective function. They also discuss the geodesic convexity (g-convexity) of the objective function, which ensures that any local minimum found is indeed a global minimum.

Datasets

Not discussed in the paper. The simulations utilize synthetic data generated from multivariate Student t-distributions and a spiked covariance model.

Key Insights and Ideas

  • Decoupling Eigenvalue and Eigenvector Shrinkage: The most significant insight is the development of a penalty that selectively shrinks eigenvalues towards a target spectrum without distorting the eigenvectors. This is crucial for exploiting known spectral structures (e.g., low-rank) without introducing unwanted biases in the orientation of the data.
  • Robustness Meets Spectral Structure: The SRTy estimator combines the robustness of Tyler's M-estimator with the ability to incorporate prior knowledge about the spectral structure of the covariance matrix. This makes it particularly powerful for applications where both outliers and specific eigenvalue distributions are present.
  • Interpretable Fixed-Point Equation: The derived fixed-point equation provides a clear mathematical structure for the estimator, making it easier to understand its behavior and properties.
  • Efficient Computation via MM Algorithm: The development of an MM algorithm ensures that the proposed estimator can be computed efficiently, even for large-dimensional problems.

Key Results

  • The paper demonstrates through simulations that the SRTy estimator converges to the fixed-point equation, empirically validating the proposed method.
  • In scenarios with heavy-tailed data (Student t-distribution), the SRTy estimator, especially when using an appropriate target spectrum, significantly outperforms standard estimators like SCM, Tyler's M-estimator (Ty), and the regularized Tyler's M-estimator (RTy) in terms of Normalized Mean Squared Error (NMSE) on both the shape matrix and the spectrum.
  • For a spiked covariance model, the SRTy estimator, particularly when the target spectrum is adaptively chosen using methods like eFusion, shows improved performance over RTy and other estimators, highlighting its ability to leverage spectral structure.
  • The paper shows that when the target spectrum $A_T$ is set to the identity matrix ($I$), the SRTy estimator reduces to existing regularized Tyler's M-estimators.

Limitations and Future Work

  • Target Spectrum Selection: While the paper explores using an auxiliary estimator (eFusion) to adaptively select the target spectrum, the question of optimal target spectrum selection remains an important area. The authors mention that this will be discussed in a forthcoming communication.
  • Existence and Uniqueness: The conditions for the existence and uniqueness of the SRTy estimator are not fully detailed in this paper due to space limitations, and are deferred to a future publication.
  • Theoretical Guarantees: While g-convexity is discussed, further theoretical analysis on the performance guarantees and asymptotic properties of the SRTy estimator could be beneficial.

Summary updated

Abstract

Covariance matrices usually exhibit specific spectral structures, such as low-rank ones in the case of factor models. In order to exploit this prior knowledge in a robust estimation process, we propose a new regularized version of Tyler's M-estimator of covariance matrix. This estimator is expressed as the minimizer of a robust M -estimating cost function plus a penalty that is unitary invariant (i.e., that only applies on the eigenvalue) that shrinks the estimated spectrum toward a fixed target. The structure of the estimate is expressed through an interpretable fixed-point equation. A majorization-minimization (MM) algorithm is derived to compute this estimator, and the g-convexity of the objective is also discussed. Several simulation studies illustrate the interest of the approach and also explore a method to automatically choose the target spectrum through an auxiliary estimator.
BibTeX
@misc{breloy2019spectral,
  author = {Arnaud Breloy and Esa Ollila and Frédéric Pascal},
  title = {{Spectral Shrinkage of Tyler's $M$-Estimator of Covariance Matrix}},
  year = {2019},
  doi = {10.1109/camsap45676.2019.9022652},
}
Figure