Large-Dimensional Behavior of Regularized Maronna's M-Estimators of Covariance Matrices

Nicolas Auguin, David Morales-Jiménez, Matthew R. McKay, Romain CouilletIEEE TSP 2018 · DOI

TL;DR

This paper investigates the behavior of robust covariance matrix estimators (M-estimators) when both the number of data samples and the dimensionality of the data grow large. The authors introduce and analyze regularized versions of these estimators, showing they offer improved performance and robustness, especially in the presence of outliers, and provide a data-driven method for selecting the optimal regularization parameter.

What Problem Does This Paper Address?

Estimating the covariance matrix is crucial in many fields, from signal processing to finance. The standard sample covariance matrix (SCM) has limitations: it's sensitive to outliers, performs poorly when the number of samples is close to the number of variables, and isn't even invertible in the under-sampled case (fewer samples than variables). While robust M-estimators (like Maronna's) handle outliers better, they still face the "data scarcity" issue. Regularized estimators (like the regularized SCM or RSCM) address data scarcity by shrinking the SCM towards a target (e.g., the identity matrix), but they inherit the SCM's sensitivity to outliers. This paper tackles the challenge of developing estimators that are both robust to outliers and perform well in high-dimensional, potentially under-sampled settings.

Methods

The core of this work lies in applying random matrix theory (RMT) to analyze the asymptotic behavior of these estimators. The authors extend previous RMT results to a broader class of regularized M-estimators.

  • Outlier-Free Analysis: They first analyze the estimators in a "clean" data scenario (no outliers). Using RMT, they show that many regularized M-estimators become asymptotically equivalent to the regularized SCM (RSCM) when the regularization parameter is optimally chosen. This means the specific form of the robust M-estimator becomes less critical for performance in this regime.
  • Robustness Analysis: To quantify robustness to outliers, they introduce a novel "measure of influence" (MI). This metric captures the bias introduced by a small fraction of contaminating data. They derive analytical expressions for this MI for both non-regularized and regularized estimators under a specific cluster contamination model.
  • Optimal Regularization: For the outlier-free case, they propose a data-driven method to find the optimal regularization parameter that minimizes the quadratic loss. This optimal parameter choice is shown to equalize the performance of different regularized M-estimators.

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and simulations using synthetically generated data based on specified covariance matrices ($C_N$ and $D_N$).

Key Insights and Ideas

  • Asymptotic Equivalence: In high dimensions, optimally regularized M-estimators behave like the regularized SCM (RSCM). This simplifies analysis and suggests that for optimal performance in clean data, the choice of the robust component (the specific M-estimator) matters less than the regularization strategy.
  • Robustness Metric: The proposed "measure of influence" provides a formal way to compare the robustness of different estimators to outliers, going beyond just comparing weights assigned to outlier samples.
  • Huber vs. Tyler: In the non-regularized case, the Huber estimator generally shows better robustness than the Tyler estimator. However, in the regularized case, the Tyler estimator can be more robust, especially when outliers are "less correlated" than the underlying data. This highlights the complex interplay between regularization and robustness.
  • Optimal Regularization is Key: The paper strongly emphasizes that properly choosing the regularization parameter is crucial for both performance (in clean data) and robustness (in contaminated data). An optimally chosen parameter can make different M-estimators achieve similar, near-optimal performance.

Key Results

  • Outlier-Free Performance: For any regularized M-estimator, there exists an optimal regularization parameter that minimizes its quadratic loss. When this parameter is chosen optimally, all these estimators achieve the same asymptotic performance, matching that of an optimally regularized RSCM.
  • Robustness Comparison:
    • In the non-regularized case, the measure of influence (MI) shows that M-estimators are generally more robust than the SCM.
  • When $Tr(C_N^{-1}D_N) > 1$, both Tyler and Huber M-estimators are more robust than the SCM.
  • When $Tr(C_N^{-1}D_N) < 1$, Tyler's estimator is less robust than Huber's and the SCM. This suggests Huber might be a safer choice when the properties of outliers are unknown.
  • In the regularized case, the Tyler estimator can be less sensitive to variations in the optimal regularization parameter compared to Huber, suggesting a potential advantage in robustness when the optimal parameter is slightly misestimated.
  • Data-Driven Parameter Selection: The paper provides a method to estimate the optimal regularization parameter, which is shown to work well in simulations.

Limitations and Future Work

  • Contamination Model: The robustness analysis is primarily based on a specific "cluster contamination" model. The behavior under other contamination models might differ.
  • "Clean-Data Optimal" Regularization: The proposed optimal regularization parameter is derived assuming clean data. The paper acknowledges that this choice might be suboptimal if a significant, non-vanishing proportion of outliers is present, especially if prior knowledge about outliers could be exploited.
  • Future Work: The authors suggest investigating different choices of regularization and estimators when prior knowledge about outliers is available. They also plan to explore the impact of more complex contamination scenarios.

Summary updated

BibTeX
@article{auguin2018large,
  author = {Nicolas Auguin and David Morales-Jiménez and Matthew R. McKay and Romain Couillet},
  title = {{Large-Dimensional Behavior of Regularized Maronna's M-Estimators of Covariance Matrices}},
  journal = {IEEE Trans. Signal Process.},
  year = {2018},
  doi = {10.1109/tsp.2018.2831629},
}
Figure