Grouping effects of sparse CCA models in variable selection

Kefei Liu, Qi Long, Li ShenarXiv 2020 · arXiv

Introduction

Canonical Correlation Analysis (CCA) is a cornerstone of multivariate statistics, designed to uncover relationships between two distinct sets of variables—for instance, linking genetic markers to brain imaging features. In the modern era of high-dimensional data, where the number of features ($p$) often dwarfs the number of samples ($n$), standard CCA becomes ill-posed. To address this, researchers employ Sparse CCA (SCCA), which incorporates $L_1$ penalties to select a small, interpretable subset of relevant variables.

However, standard SCCA is computationally demanding because it requires handling complex covariance matrices. A popular alternative is "simplified SCCA," which assumes variable independence by treating covariance matrices as diagonal. While computationally efficient, the theoretical implications of this simplification on variable selection—specifically how it handles highly correlated groups of variables—remained largely unexplored until now.

Theoretical Insights: Grouping vs. Lone Wolf Behavior

This research investigates the fundamental differences between standard SCCA, which maximizes canonical correlation, and simplified SCCA, which maximizes canonical covariance. Through an analysis of Karush-Kuhn-Tucker (KKT) optimization conditions, the authors demonstrate that these two models exhibit divergent behaviors when encountering clusters of correlated variables.

The Grouping Effect of Simplified SCCA

Simplified SCCA ignores the covariance between variables within its constraints. Geometrically, this results in a feasible region that allows for a "grouping effect." If a set of variables is highly correlated and relevant to the outcome, simplified SCCA will select the entire group and assign them similar weights. This is ideal for biological applications where researchers want to identify entire pathways or functional brain networks.

Standard SCCA as a "Lone Wolf"

In contrast, standard SCCA accounts for the correlation between variables. When faced with a group of redundant, highly correlated features, the model tends to select only one or two "dominant" variables to represent the group, driving the weights of the remaining correlated variables to zero. This "lone wolf" behavior is preferable when the goal is to find the absolute minimum number of unique biomarkers.

Geometric interpretation of the grouping effect: (a) $L_1$ constraint, (b) simplified SCCA constraint allowing for a range of solutions (grouping), and (c) standard SCCA constraint leading to a unique representative solution.

Methods and Algorithmic Extensions

The authors developed several tools to facilitate this comparison:

  • Algorithmic Refinement: They extended a foundational mathematical lemma to solve the simplified SCCA subproblem across a broader range of sparsity parameters.
  • New Solvers: They introduced greedy algorithms for computing multiple canonical components using both data and covariance matrix deflation. Additionally, a Linearized Alternating Direction Method of Multipliers (ADMM) was provided to fit the standard SCCA model efficiently.

Experimental Results

The models were tested on both synthetic and real-world datasets to validate the theoretical findings.

Synthetic Data Performance

In Setup 1 (uncorrelated variables), standard SCCA outperformed the simplified version, achieving a higher F1 score (0.413 vs. 0.358) and better test-set correlation. However, in Setup 2 (variables organized into 20 highly correlated groups), the simplified SCCA dominated. By successfully recovering entire groups, it achieved an F1 score of 0.751, nearly double that of standard SCCA (0.378).

Comparison of recovered weight vectors: The simplified SCCA (middle/bottom rows) clearly recovers the block structures of correlated variables, whereas standard SCCA (middle/bottom rows) selects only sparse representatives within those blocks.

Real-World Application: Alzheimer’s Disease (ADNI)

Using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI), the authors compared 1,542 genetic markers against 116 brain imaging features. Both models identified the well-known APOE risk gene. However, the selection patterns mirrored the theory: simplified SCCA identified a large cluster of SNPs within the APOE gene and over 60 correlated brain features, while standard SCCA selected only a few dominant SNPs and approximately 20 imaging features.

Conclusion

This study provides a theoretical framework for choosing between SCCA variants. Simplified SCCA is not merely a "cheap shortcut"; it is a specialized tool for identifying grouped variables, making it highly suitable for multi-omics and neuroimaging data where features naturally cluster. Conversely, standard SCCA remains the superior choice for high-precision variable selection in datasets with low redundancy.

Summary updated

Abstract

The sparse canonical correlation analysis (SCCA) is a bi-multivariate association model that finds sparse linear combinations of two sets of variables that are maximally correlated with each other. In addition to the standard SCCA model, a simplified SCCA criterion which maixmizes the cross-covariance between a pair of canonical variables instead of their cross-correlation, is widely used in the literature due to its computational simplicity. However, the behaviors/properties of the solutions of these two models remain unknown in theory. In this paper, we analyze the grouping effect of the standard and simplified SCCA models in variable selection. In high-dimensional settings, the variables often form groups with high within-group correlation and low between-group correlation. Our theoretical analysis shows that for grouped variable selection, the simplified SCCA jointly selects or deselects a group of variables together, while the standard SCCA randomly selects a few dominant variables from each relevant group of correlated variables. Empirical results on synthetic data and real imaging genetics data verify the finding of our theoretical analysis.
BibTeX
@misc{liu2020grouping,
  author = {Kefei Liu and Qi Long and Li Shen},
  title = {{Grouping effects of sparse CCA models in variable selection}},
  year = {2020},
  eprint = {2008.03392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.03392},
}
Figure