Tuning-Free Structured Sparse PCA via Deep Unfolding Networks

Long Chen, Xianchao Xiu

Abstract

Sparse principal component analysis (PCA) is a well-established dimensionality reduction technique that is often used for unsupervised feature selection (UFS). However, determining the regularization parameters is rather challenging, and conventional approaches, including grid search and Bayesian optimization, not only bring great computational costs but also exhibit high sensitivity. To address these limitations, we first establish a structured sparse PCA formulation by integrating $\ell_1$-norm and $\ell_{2,1}$-norm to capture the local and global structures, respectively. Building upon the off-the-shelf alternating direction method of multipliers (ADMM) optimization framework, we then design an interpretable deep unfolding network that translates iterative optimization steps into trainable neural architectures. This innovation enables automatic learning of the regularization parameters, effectively bypassing the empirical tuning requirements of conventional methods. Numerical experiments on benchmark datasets validate the advantages of our proposed method over the existing state-of-the-art methods.

Introduction

Unsupervised Feature Selection (UFS) is a critical task in high-dimensional data analysis, helping researchers identify the most relevant features when labels are unavailable. While Sparse PCA is a popular tool for this, it faces two major hurdles in practical application. First is the "tuning nightmare": traditional Sparse PCA relies on regularization parameters (like $\lambda$) to control sparsity, and finding the optimal values usually requires exhaustive grid searches or complex Bayesian optimization. Second is "structural blindness": most standard methods use the $\ell_1$-norm for element-wise sparsity, failing to capture the group-wise or structural sparsity often found in real-world data.

To overcome these challenges, we introduce SPCA-Net, a novel deep unfolding network that automates hyperparameter tuning. By "unfolding" a classical optimization algorithm into a neural network, SPCA-Net learns optimal regularization parameters directly from data, achieving state-of-the-art performance with significantly less manual effort.

Methodology

Structured Sparse PCA Formulation

Our approach begins with a new mathematical model that integrates both the $\ell_1$-norm (for local, element-wise sparsity) and the $\ell_{2,1}$-norm (for global, row-wise sparsity). This structured approach allows the model to capture both fine-grained details and broader group patterns in the data, providing a more robust representation for feature selection.

Deep Unfolding (SPCA-Net)

Instead of solving the optimization problem with a traditional iterative solver like the Alternating Direction Method of Multipliers (ADMM), we translate the ADMM steps into a neural network architecture.

Architecture of SPCA-Net: Translating ADMM optimization steps into a multi-stage deep unfolding network.

In this framework, each "stage" of the network corresponds to one full iteration of the ADMM algorithm. Parameters that are usually hand-tuned—such as regularization weights $\lambda, \mu$ and penalty parameters $\alpha, \beta$—are treated as trainable weights. The network is composed of specific blocks (X, Y, Z, $\Lambda$, $\Pi$) that mirror the mathematical updates of the original optimization algorithm, ensuring the model remains mathematically interpretable rather than a "black box."

Experimental Results

We evaluated SPCA-Net on four diverse benchmark datasets: COIL20 (object images), Isolet (speech signals), UMIST (handwritten digits), and MSTAR (biological data). The model was compared against several state-of-the-art (SOTA) methods, including LapScore, UDFS, and SPCAFS.

Comparison of Normalized Mutual Information (NMI) across four benchmark datasets. SPCA-Net (red line) consistently achieves superior performance compared to state-of-the-art methods.

Key Findings

  • Superior Performance: SPCA-Net consistently outperformed SOTA methods across primary metrics, including Clustering Accuracy (ACC) and Normalized Mutual Information (NMI). For instance, on the Isolet dataset, SPCA-Net achieved a clustering accuracy of 59.17%, outperforming the second-best method by over 5%.
  • Efficiency: While traditional optimization might take hundreds of iterations to converge, SPCA-Net achieves superior results in just 3 to 5 stages, drastically reducing computational overhead during inference.
  • Tuning-Free Automation: By training the network end-to-end, the model automatically finds the best hyperparameters for a given dataset, bypassing the need for manual trial-and-error.

Discussion and Conclusion

SPCA-Net demonstrates that deep unfolding can effectively bridge the gap between classical optimization and deep learning. By embedding the structure of ADMM into a neural network, we achieve a model that is both interpretable and highly efficient.

One limitation noted is "stage sensitivity": performance can degrade if the network is too deep (e.g., more than 10 stages), suggesting potential overfitting. Future work will explore more effective loss functions and investigate "deep equilibrium networks" to potentially find the infinite-depth limit of these unfolding stages more efficiently. Our code is available at https://github.com/xianchaoxiu/SPCA-Net.

Summary updated

Abstract

Sparse principal component analysis (PCA) is a well-established dimensionality reduction technique that is often used for unsupervised feature selection (UFS). However, determining the regularization parameters is rather challenging, and conventional approaches, including grid search and Bayesian optimization, not only bring great computational costs but also exhibit high sensitivity. To address these limitations, we first establish a structured sparse PCA formulation by integrating $\ell_1$-norm and $\ell_{2,1}$-norm to capture the local and global structures, respectively. Building upon the off-the-shelf alternating direction method of multipliers (ADMM) optimization framework, we then design an interpretable deep unfolding network that translates iterative optimization steps into trainable neural architectures. This innovation enables automatic learning of the regularization parameters, effectively bypassing the empirical tuning requirements of conventional methods. Numerical experiments on benchmark datasets validate the advantages of our proposed method over the existing state-of-the-art methods. Our code will be accessible at https://github.com/xianchaoxiu/SPCA-Net.
BibTeX
@misc{chen2025tuning,
  author = {Long Chen and Xianchao Xiu},
  title = {{Tuning-Free Structured Sparse PCA via Deep Unfolding Networks}},
  year = {2025},
  eprint = {2502.20837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.20837},
}
Figure