Ultrahigh-Dimensional Robust and Efficient Sparse Regression Using Non-Concave Penalized Density Power Divergence

Abhik Ghosh, Subhabrata MajumdarIEEE Trans. Information Theory 2020 · OpenReview

TL;DR

This paper introduces a new method for sparse regression in ultrahigh-dimensional settings that is both robust to data contamination and efficient. By combining non-concave penalties with the density power divergence (DPD) loss function, the authors develop an estimator that can simultaneously perform variable selection and robust parameter estimation, outperforming existing methods in simulations, especially under noisy data.

What Problem Does This Paper Address?

In high-dimensional data analysis, where the number of features (p) can be much larger than the number of observations (n), sparse regression techniques are crucial for selecting relevant variables and building interpretable models. Traditional methods like LASSO, while effective for sparsity, are sensitive to outliers and data contamination. Existing robust regression methods often rely on L1-penalization, and their theoretical properties in ultrahigh dimensions are not fully understood. This paper addresses the need for a method that can handle both ultrahigh dimensionality and data contamination effectively, ensuring reliable variable selection and parameter estimation.

Methods

The core of this work is the development of the Minimum Non-concave Penalized DPD (MNPDPDE) estimator. This method combines two key ideas:

  • Density Power Divergence (DPD) Loss Function: This loss function is known for its robustness against infinitesimal contamination, offering a trade-off between robustness and efficiency controlled by a tuning parameter $\alpha$.
  • Non-concave Penalties: Penalties like SCAD and MCP are used to induce sparsity and achieve variable selection consistency, overcoming the bias issues associated with L1-penalization in standard regression.

The authors propose an alternating algorithm based on the Concave-Convex Procedure (CCCP) to efficiently compute the MNPDPDE. They also provide theoretical guarantees for their estimator, including influence function analysis for robustness and proofs of large-sample oracle properties in ultrahigh dimensions.

Datasets

The paper does not introduce new datasets. Instead, it relies on simulated data generated according to specific experimental setups to evaluate the performance of their proposed method against existing ones. These simulations involve varying signal strengths, dimensions, and introducing different types of contamination (Y-outliers and X-outliers) to assess robustness and efficiency.

Key Insights and Ideas

  • Synergy of DPD and Non-concave Penalties: The paper demonstrates that combining the robustness of DPD with the sparsity-inducing properties of non-concave penalties leads to a powerful estimator for ultrahigh-dimensional sparse regression.
  • Robustness to Contamination: The DPD loss function, with $\alpha > 0$, provides a significant advantage in handling data contamination, a common issue in real-world datasets. This robustness is theoretically supported by influence function analysis.
  • Oracle Properties in Ultrahigh Dimensions: The authors prove that their MNPDPDE estimator achieves desirable large-sample oracle properties, meaning it can perform variable selection and parameter estimation as well as if the true sparse model were known in advance, even in ultrahigh-dimensional settings.
  • Algorithmic Efficiency: The proposed CCCP-based algorithm is shown to be effective for computing the MNPDPDE, providing a practical solution for high-dimensional problems.

Key Results

The simulation studies, conducted for $p=100, 200, 500$, show that the proposed MNPDPDE method generally outperforms existing robust and non-robust methods across various metrics (MSEE, RMSPE, EE, TP, TN, MS).

  • In most scenarios, especially with data contamination, the MNPDPDE achieved the best performance, followed by LDPD-Lasso and RLARS.
  • The method demonstrated good variable selection (high TP and TN) and parameter estimation accuracy (low MSEE, RMSPE, EE).
  • The robustness of the method was evident, with performance degradation being less severe compared to other methods when outliers were present.
  • The choice of the tuning parameter $\alpha$ in the DPD loss function influences the trade-off between robustness and efficiency, with smaller $\alpha$ values (e.g., 0.4-0.6) often yielding the best results in simulations.

Limitations and Future Work

  • Choice of Tuning Parameter $\alpha$: While the paper provides theoretical insights into the role of $\alpha$, selecting an optimal $\alpha$ in practice can be challenging, especially when the extent of contamination is unknown. The authors suggest empirical methods and future work on data-driven choices for $\alpha$.
  • Error Distribution Assumptions: The theoretical analysis primarily assumes normal error distributions, although the DPD framework is more general. Extending the theory to heavy-tailed error distributions is suggested as future work.
  • Computational Complexity: While the CCCP algorithm is efficient, further exploration of its convergence properties and potential for optimization is mentioned.
  • Generalizations: The authors suggest extending their framework to graphical models and incorporating group penalties for structured sparsity, which are areas of active research. They also propose exploring more theoretically grounded ways of choosing $\alpha$ that account for model selection performance.

Summary updated

Abstract

We propose a sparse regression method based on the non-concave penalized density power divergence loss function which is robust against infinitesimal contamination in very high dimensionality. Present methods of sparse and robust regression are based on $\ell _{1}$ -penalization, and their theoretical properties are not well-investigated. In contrast, we use a general class of folded concave penalties that ensure sparse recovery and consistent estimation of regression coefficients. We propose an alternating algorithm based on the Concave-Convex procedure to obtain our estimate, and demonstrate its robustness properties using influence function analysis. Under some conditions on the fixed design matrix and penalty function, we prove that this estimator possesses large-sample oracle properties in an ultrahigh-dimensional regime. The performance and effectiveness of our proposed method for parameter estimation and prediction compared to state-of-the-art are demonstrated through simulation studies.
BibTeX
@article{ghosh2020ultrahigh,
  author = {Abhik Ghosh and Subhabrata Majumdar},
  title = {{Ultrahigh-Dimensional Robust and Efficient Sparse Regression Using Non-Concave Penalized Density Power Divergence}},
  journal = {IEEE Transactions on Information Theory},
  year = {2020},
  url = {https://openreview.net/forum?id=JiHGnP23Za},
}
Figure