On the Convergence of Maronna's M-Estimators of Scatter

Yacine Chitour, Romain Couillet, Frédéric Pascal

TL;DR

This paper provides a new, more general proof for the uniqueness of Maronna's M-estimators of scatter. Crucially, it demonstrates that Maronna's estimators, under certain conditions, converge to a specific type of Tyler's M-estimator, offering a theoretical justification for Tyler's estimator's robustness.

What Problem Does This Paper Address?

In robust statistics, estimating the scatter (covariance) of a dataset is crucial. Maronna's M-estimators offer a robust alternative to traditional methods like Maximum Likelihood Estimation (MLE), which are sensitive to outliers. While the existence and uniqueness of Maronna's estimators have been established, recent work in high-dimensional statistics (where the number of features m and samples N both grow large) has motivated new proofs. This paper addresses two main points: providing a more general proof of uniqueness for Maronna's estimators and, more significantly, establishing a connection between Maronna's and Tyler's M-estimators by showing that Maronna's estimators converge to a Tyler's estimator as a parameter t goes to zero. This convergence is particularly interesting because Tyler's estimator is known to be highly robust, and this result helps explain why.

Methods

The authors develop a novel proof strategy. They introduce a functional H(t, M) defined over the space of positive definite matrices PSDm. They show that the solutions to Maronna's M-estimator equation for a given t correspond to the critical points of this functional. By analyzing the properties of H(t, M), specifically its boundedness and behavior at the boundaries of PSDm, they prove the existence and uniqueness of a solution M(t) for any t > 0 (Item A of Theorem II.2).

The core of their contribution lies in demonstrating the convergence of M(t) to a Tyler's M-estimator Mo as t approaches zero (Item B of Theorem II.2). This is achieved by leveraging the implicit function theorem and analyzing the behavior of the estimator in the limit t → 0. They also introduce a specific weight function ū(t, x) which simplifies the analysis and shows that in this case, the convergence is to a scaled version of the Tyler's estimator.

Datasets

Not discussed in the paper. The paper focuses on theoretical properties and uses simulated data for illustration.

Key Insights and Ideas

  • Unified Proof for Uniqueness: The paper offers a more general proof for the uniqueness of Maronna's M-estimators, applicable to a wider range of well-behaved sample sets.
  • Maronna to Tyler Convergence: The most significant finding is the demonstration that Maronna's M-estimators converge to a Tyler's M-estimator as a parameter t (related to the tail behavior of the underlying distribution) goes to zero. This provides a theoretical bridge between these two robust estimation families.
  • Theoretical Justification for Tyler's Robustness: This convergence helps explain why Tyler's M-estimator is considered highly robust. As t → 0, the underlying distribution is effectively assumed to have heavier tails, and Tyler's estimator emerges as the limit of robust estimators designed for such distributions.
  • Connection to Heavy-Tailed Distributions: The parameter t in the weight function u(t, x) can be interpreted as related to the tail heaviness of the data distribution. The convergence as t → 0 means that as the assumed distribution becomes more heavy-tailed, the Maronna estimator approaches the Tyler estimator.

Key Results

  • Theorem II.2 (Item A): For any t > 0, Maronna's M-estimator equation admits a unique solution M(t) for admissible families of vectors.
  • Theorem II.2 (Item B): Under additional smoothness conditions on the weight function u, the unique solution M(t) converges to a unique Tyler's M-estimator Mo as t → 0.
  • Figure 1: Illustrates the convergence of M(t) to Mo for a specific case (Student-t distribution) using mean squared error.

Limitations and Future Work

  • Admissibility Constraint: The proofs rely on an "admissible family" of vectors, which requires linear independence of any subset of size m. While this is a mild constraint, it's a condition for the theoretical results.
  • Weight Function Smoothness: The convergence result (Item B of Theorem II.2) requires the weight function u to be twice continuously differentiable and satisfy an additional condition (U3).
  • Future Work: The authors suggest that these results are important for understanding robust M-estimation in the large dimensional (random matrix) regime and may find interest in further studies of Tyler's M-estimator in such contexts. The connection to heavy-tailed distributions and the theoretical motivation for Tyler's estimator's robustness are highlighted as key implications.

Summary updated

Abstract

In this paper, {we propose an alternative proof for the uniqueness} of Maronna's $M$-estimator of scatter (Maronna, 1976) for $N$ vector observations $\mathbf y_1,...,\mathbf y_N\in\mathbb R^m$ under a mild constraint of linear independence of any subset of $m$ of these vectors. This entails in particular almost sure uniqueness for random vectors $\mathbf y_i$ with a density as long as $N>m$. {This approach allows to establish further relations that demonstrate that a properly normalized Tyler's $M$-estimator of scatter (Tyler, 1987) can be considered as a limit of Maronna's $M$-estimator. More precisely, the contribution is to show that each $M$-estimator converges towards a particular Tyler's $M$-estimator.} These results find important implications in recent works on the large dimensional (random matrix) regime of robust $M$-estimation.
BibTeX
@article{chitour2015convergence,
  author = {Yacine Chitour and Romain Couillet and Frédéric Pascal},
  title = {{On the Convergence of Maronna's M-Estimators of Scatter}},
  journal = {IEEE Signal Processing Letters},
  year = {2015},
  doi = {10.1109/lsp.2014.2367547},
}
Figure