Robust Model Order Selection in Large Dimensional Elliptically Symmetric Noise

Eugénie Terreaux, Jean‐Philippe Ovarlez, Frédéric PascalarXiv 2017 · DOI · arXiv

Abstract

This paper deals with model order selection in context of correlated noise. More precisely, one considers sources embedded in an additive Complex Elliptically Symmetric (CES) noise, with unknown parameters. The main difficultly for estimating the model order lies into the noise correlation, namely the scatter matrix of the corresponding CES distribution. In this work, to tackle that problem, one adopts a two-step approach: first, we develop two different methods based on a Toeplitz-structured model for estimating this unknown scatter matrix and for whitening the correlated noise. Then, we apply Maronna's $M$-estimators on the whitened signal to estimate the covariance matrix of the "decorrelated" signal in order to estimate the model order. The proposed methodology is based both on robust estimation theory as well as large Random Matrix Theory, and original results are derived, proving the efficiency of this methodology. Indeed, the main theoretical contribution is to derive consistent robust estimators for the covariance matrix of the signal-plus-correlated noise in a large dimensional regime and to propose efficient methodology to estimate the rank of signal subspace. Finally, as shown in the analysis, these results show a great improvement compared to the state-of-the-art, on both simulated and real hyperspectral images.

Introduction

In advanced signal processing applications such as radar or hyperspectral imaging, determining the number of signal sources—known as model order selection—is a critical first step. While classical tools like the Akaike Information Criterion (AIC) or Minimum Description Length (MDL) are standard for Gaussian, uncorrelated ("white") noise, they often fail in real-world scenarios. Modern datasets are typically high-dimensional, spatially correlated, and non-Gaussian, often following heavy-tailed Complex Elliptically Symmetric (CES) distributions. Under these conditions, traditional methods tend to drastically overestimate the number of sources, leading to poor downstream performance.

Methodology: A Two-Step Robust Framework

To address these challenges, the authors propose a "whiten-then-estimate" framework that leverages both robust statistics and Random Matrix Theory (RMT).

1. Robust Whitening with Toeplitz Structure

Since the noise correlation (the scatter matrix) is usually unknown, the authors assume it possesses a Toeplitz structure, a common property for stationary signals. They introduce two estimation methods: a "toeplitzified" Sample Covariance Matrix (SCM) and a "toeplitzified" Fixed-Point (FP) estimator. This step effectively removes spatial correlation, "whitening" the background noise.

2. Robust Covariance Estimation and RMT Thresholding

After whitening, the signal is processed using Maronna’s $M$-estimator. This robust tool weights observations based on their distance from the mean, neutralizing the "texture" or non-Gaussian spikes inherent in CES noise.

Eigenvalue distribution of the estimated covariance matrix under heavy-tailed noise. The red line indicates the theoretical RMT threshold.

The final step involves analyzing the eigenvalues of this robust matrix. Using RMT, the authors derive a theoretical threshold ($t$) based on the Marchenko-Pastur law. In this high-dimensional regime ($N, m \to \infty$), eigenvalues belonging to the noise floor stay below this threshold, while those corresponding to signal sources appear as outliers above it.

Key Results and Insights

The methodology was rigorously tested on both synthetic CES noise (Inverse Gamma and Student-t distributions) and real hyperspectral datasets like Indian Pines and SalinasA.

  • Detection Performance: In simulations, the proposed Fixed-Point (FP) method correctly identified signal sources at SNRs as low as -5dB, whereas classical AIC failed even at high SNRs.
  • Real-World Accuracy: On the Indian Pines image (ground truth $\approx$ 16 sources), the proposed methods estimated 11–12 sources. In stark contrast, classical AIC and non-whitened robust methods estimated over 210 sources, demonstrating that whitening is essential for accuracy in correlated environments.
Detection of a signal source: the eigenvalue corresponding to the signal successfully crosses the RMT threshold $t$ after robust whitening and estimation.

Conclusion

By combining Toeplitz-structured whitening with robust $M$-estimation and RMT-based thresholding, this framework provides a stable and theoretically grounded approach to model order selection. The results confirm that stripping away noise "texture" and correlation allows RMT tools designed for Gaussian noise to be successfully applied to complex, non-Gaussian real-world data. Future work may focus on refining the estimation of unknown noise parameters and extending the framework to non-stationary noise environments.

Summary updated

Abstract

This paper deals with model order selection in context of correlated noise. More precisely, one considers sources embedded in an additive Complex Elliptically Symmetric (CES) noise, with unknown parameters. The main difficultly for estimating the model order lies into the noise correlation, namely the scatter matrix of the corresponding CES distribution. In this work, to tackle that problem, one adopts a two-step approach: first, we develop two different methods based on a Toeplitz-structured model for estimating this unknown scatter matrix and for whitening the correlated noise. Then, we apply Maronna's $M$-estimators on the whitened signal to estimate the covariance matrix of the "decorrelated" signal in order to estimate the model order. The proposed methodology is based both on robust estimation theory as well as large Random Matrix Theory, and original results are derived, proving the efficiency of this methodology. Indeed, the main theoretical contribution is to derive consistent robust estimators for the covariance matrix of the signal-plus-correlated noise in a large dimensional regime and to propose efficient methodology to estimate the rank of signal subspace. Finally, as shown in the analysis, these results show a great improvement compared to the state-of-the-art, on both simulated and real hyperspectral images.
BibTeX
@misc{terreaux2017robust,
  author = {Eugénie Terreaux and Jean‐Philippe Ovarlez and Frédéric Pascal},
  title = {{Robust Model Order Selection in Large Dimensional Elliptically Symmetric Noise}},
  year = {2017},
  eprint = {1710.06735},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1710.06735},
}
Figure