Tuning-parameter selection in regularized estimations of large covariance matrices

Yixin Fang, Binhuan Wang, Yang FengJournal of Statistical Computation and Simulation 2013 · arXiv

TL;DR

This paper tackles the crucial but underspecified problem of selecting tuning parameters for regularized covariance matrix estimators. The authors provide concrete, data-driven recommendations for choosing the number of folds in cross-validation, distinguishing between Frobenius and operator norm accuracy measures. They also demonstrate that cross-validation generally outperforms bootstrap methods for this task.

What Problem Does This Paper Address?

Estimating covariance matrices is fundamental in many statistical analyses, especially when dealing with high-dimensional data where the number of variables ($p$) is large relative to the number of samples ($n$). Regularization techniques are essential to obtain stable and meaningful estimates in such scenarios. However, these methods introduce tuning parameters that significantly impact performance. While cross-validation is a common approach for selecting these parameters, there's a notable lack of consensus on how to best configure it, specifically regarding the number of "folds" (data partitions). This paper aims to fill this gap by providing empirical guidelines for optimal cross-validation strategies and comparing them against bootstrap-based methods.

Methods

The core of this paper lies in a comprehensive simulation study comparing various tuning parameter selection methods. The authors focus on four types of regularized estimators: banding, tapering, hard-thresholding, and soft-thresholding. They evaluate two primary selection strategies:

  • Cross-Validation (CV): They investigate V-fold CV, where the data is split into V folds. One fold is used for validation, and the remaining V-1 folds are used for training. They specifically explore different values of V (2, 3, 5, 10) and also consider "reverse" CV where the validation set is larger than the training set.
  • Bootstrap: They implement bootstrap methods to estimate the risk associated with different tuning parameters.

The performance of these selection methods is assessed by measuring the estimation accuracy of the resulting covariance matrices using two common norms: the Frobenius norm (sum of squared differences of all elements) and the operator norm (maximum singular value of the difference matrix).

Datasets

The study uses simulated datasets generated from three distinct covariance models, each with varying parameters. These models are adopted from a previous work (Yi and Zou, 2012) to ensure comparability. The simulations consider four settings of sample size ($n$) and dimension ($p$): (100, 100), (100, 200), (200, 100), and (200, 200). No new datasets or benchmarks are introduced; the focus is on evaluating existing methods under controlled conditions.

Key Insights and Ideas

  • Norm Matters for CV Folds: The optimal number of folds for cross-validation is highly dependent on the chosen accuracy norm.
  • For Frobenius norm accuracy, a larger training set is preferred, suggesting higher fold numbers (e.g., 10-fold CV).
  • For operator norm accuracy, a larger validation set is beneficial, pointing towards lower fold numbers (e.g., 2-fold or reverse 3-fold CV).
  • CV Generally Outperforms Bootstrap: Across various scenarios, cross-validation methods, particularly those with "optimal" fold choices, consistently yield better results than bootstrap-based tuning parameter selection.
  • Banding/Tapering vs. Thresholding: The banding and tapering estimators generally perform better than thresholding estimators when the true covariance matrix has a corresponding structure, highlighting the importance of model-based regularization.

Key Results

  • Frobenius Norm: 10-fold cross-validation was found to be the best performing method for all models and estimators when accuracy is measured by the Frobenius norm. 5-fold CV performed comparably.
  • Operator Norm: For operator norm accuracy, reverse 3-fold CV was often best for banding and tapering estimators. For thresholding estimators, 2-fold CV (or 2-fold CV with random splits) performed best. This contrasts sharply with the Frobenius norm findings.
  • Bootstrap vs. CV: Bootstrap methods performed similarly to 10-fold CV for banding and tapering estimators in the Frobenius norm setting. However, for the operator norm, reverse 3-fold CV was recommended over bootstrap due to comparable performance and significantly lower computational cost.

Limitations and Future Work

  • Computational Cost: While cross-validation is generally preferred, some variants (especially higher-fold CV) can be computationally intensive. The paper notes that bootstrap is more computationally expensive than reverse 3-fold CV.
  • Approximation for Operator Norm: The paper derives a "rough approximation" for the operator risk estimation using bootstrap, acknowledging that this approximation is less accurate for high-dimensional data due to the curse of dimensionality.
  • Future Work: The authors suggest that their work on approximating operator risk could stimulate more accurate approximations. They also implicitly suggest further investigation into the theoretical underpinnings of why certain CV fold choices are optimal for different norms.

Summary updated

Abstract

Recently many regularized estimators of large covariance matrices have been proposed, and the tuning parameters in these estimators are usually selected via cross-validation. However, there is a lack of consensus on the number of folds for conducting cross-validation. One round of cross-validation involves partitioning a sample of data into two complementary subsets, a training set and a validation set. In this manuscript, we demonstrate that if the estimation accuracy is measured in the Frobenius norm, the training set should consist of majority of the data; whereas if the estimation accuracy is measured in the operator norm, the validation set should consist of majority of the data. We also develop methods for selecting tuning parameters based on the bootstrap and compare them with their cross-validation counterparts. We demonstrate that the cross-validation methods with ‘optimal’ choices of folds are more appropriate than their bootstrap counterparts.
BibTeX
@article{fang2013tuning,
  author = {Yixin Fang and Binhuan Wang and Yang Feng},
  title = {{Tuning-parameter selection in regularized estimations of large covariance matrices}},
  journal = {Journal of Statistical Computation and Simulation},
  year = {2013},
}
Figure