Sparse regression frameworks, most notably the Lasso, are widely celebrated for their ability to perform simultaneous variable selection and estimation. By adding an $\ell_1$ penalty to the loss function, these models compress coefficients toward zero, effectively selecting a parsimonious subset of predictors.
However, this elegant mechanism breaks down when faced with highly correlated or collinear features. In the presence of multicollinearity, standard sparse selection algorithms exhibit extreme instability, frequently making arbitrary choices that compromise both scientific interpretability and model reliability.
The Geometry of Arbitrary Selection
To understand why standard sparse selection fails under correlation, we must look at the geometry of the optimization landscape.
When predictors are independent, the contours of the least-squares loss function are spherical or well-conditioned ellipses. The sharp corners of the $\ell_1$ bounding ball (the diamond) intersect these contours at precise, unique points along the coordinate axes, yielding a stable and deterministic sparse solution.
Independent Features: Correlated Features:
▲ ▲
│ ▲ │ ___---───---___
│ ┌─┴─┐ │ ┌─▀─┐ ▀─┐
│ │ ◊ │ │ │ ◊ │ │
───────┼─┴───┴─► ───────┼──┴───┴───────────────►
│ │ │ │ │ │ │
│ └─┬─┘ │ └─▄─┘ ▄─┘
│ ▼ │ ▀▀▀---───---▀▀▀
(Unique, stable intersection) (Flat edge alignment: arbitrary choice)
When two features are highly correlated, the loss contours stretch into highly elongated, flat ellipses. If the correlation is strong enough, the edge of the loss contour aligns almost perfectly parallel to the flat face of the $\ell_1$ diamond. This alignment creates a flat optimization plateau where many different combinations of the two features yield virtually identical loss values.
As a result, sparse regression models tend to arbitrarily select a single representative feature from a group of highly correlated predictors while zeroing out the rest, a phenomenon detailed by Satoshi Hara et al. in Enumerate Lasso Solutions for Feature Selection. Because the optimization landscape is so flat, minor data perturbations or trivial amounts of numerical noise can cause the optimal solution to jump violently from one feature to another. This leads to low stability across different subsets of the same dataset, as shown by Bo Xin et al. in Efficient Generalized Fused Lasso and its Application to the Diagnosis of Alzheimer's Disease.
Why Instability Sabotages Interpretability
This arbitrary selection behavior is not merely a mathematical quirk; it severely undermines the utility of machine learning in domain-specific research.
In genomics, for instance, genes often operate within tightly synchronized pathways or exhibit high linkage disequilibrium. When standard sparse regression is applied to such data, it picks an isolated gene at random and discards its highly correlated neighbors. As Matt J. Silver et al. demonstrate in Pathways-Driven Sparse Regression Identifies Pathways and Genes Associated with High-Density Lipoprotein Cholesterol in Two Asian Cohorts, this behavior obscures the full biological context, hiding entire sets of associated variants that are functionally relevant to the disease under study.
A similar failure mode occurs in neuroimaging. In Small-sample brain mapping: sparse recovery on spatially correlated designs with randomization and clustering, Gael Varoquaux et al. observe that standard sparse recovery produces fragmented, scattered voxel maps. Because neighboring voxels in fMRI scans are highly correlated, the model selects isolated points rather than identifying the true, spatially contiguous brain regions responsible for cognitive tasks. This lack of structural coherence is also highlighted by Hongbao Cao et al. in Integrating fMRI and SNP data for biomarker identification for schizophrenia with a sparse representation based variable selection method.
This challenge is not confined to linear models. Ensembles and deep architectures face identical hurdles when extracting sparse explanations. In Interpretable Random Forests via Rule Extraction, Gérard Biau notes that rule-extraction techniques for random forests become highly unstable when features are redundant, generating completely different rule sets across minor splits of the data.
Grouping Mechanisms: Forcing Cohesive Selection
To combat arbitrary selection, one major research direction modifies the penalty geometry to force highly correlated features to be selected or discarded together as a cohesive unit.
Automatic Feature Grouping
When structural relationships are not known in advance, algorithms must discover groups dynamically. In Efficient Sparse Modeling with Automatic Feature Grouping, Leon Wenliang Zhong leverages the OSCAR (Octagonal Selection and Clustering Algorithm for Regression) penalty, which combines an $\ell_1$ penalty with a pairwise $\ell_\infty$ penalty. The $\ell_\infty$ component penalizes differences between pairs of coefficients, effectively forcing highly correlated variables to share identical coefficient values.
Similarly, Lei Han et al. propose Discriminative Feature Grouping, which utilizes a pairwise bounding constraint to group correlated variables while maintaining sharp boundaries between distinct, non-correlated feature clusters.
Graph-Guided Regularization
When prior domain knowledge is available, it can be encoded directly into the optimization problem via a feature graph. In this setup, nodes represent features and edges represent known correlations or functional relationships.
In Stabilizing High-Dimensional Prediction Models Using Feature Graphs, Shivapratap Gopakumar et al. apply Laplacian-based graph regularization to electronic medical records. By penalizing differences between connected nodes, the model ensures that related medical codes receive similar weights, stabilizing the model against the arbitrary omissions typical of standard Lasso.
Sorted and Ordered Penalties: SLOPE and OWL
A particularly powerful mathematical evolution in this field is the development of sorted penalties, which group correlated variables without requiring predefined clusters or pairwise constraints.
In Ordered Weighted L1 Regularized Regression with Strongly Correlated Covariates: Theoretical Aspects, Mário A. T. Figueiredo et al. provide the theoretical foundation for the Ordered Weighted $\ell_1$ (OWL) regularizer. Instead of applying a uniform weight to all coefficients, OWL sorts the absolute values of the coefficients in descending order before applying a sequence of decreasing weights $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p \ge 0$.
$$\text{OWL Penalty}(\beta) = \sum_{i=1}^p \lambda_i |\beta|_{(i)}$$
Where $|\beta|_{(1)} \ge |\beta|_{(2)} \ge \dots \ge |\beta|_{(p)}$ are the ordered magnitudes.
This sorting mechanism fundamentally alters the geometry of the constraint ball. When two features are highly correlated, assigning them different weights would incur a higher penalty than assigning them equal, averaged weights. Consequently, OWL automatically clusters highly correlated variables, assigning them identical coefficients.
Building on this concept, Fabio Feser et al. introduce Sparse-group SLOPE: adaptive bi-level selection with FDR-control, demonstrating that sorted penalties can maintain strict false discovery rate control and selection consistency even under intense multicollinearity. To make these methods practical, Xavier Dupuis et al. map out the exact continuous, piecewise linear solution path in The Solution Path of SLOPE, allowing researchers to efficiently locate optimal regularization parameters.
Bypassing the Single-Model Search
Rather than trying to engineer a single, perfect penalty function that balances sparsity and grouping, an alternative paradigm rethinks the selection process entirely by expanding the search space or altering the statistical framework.
Multi-Solution Enumeration
If the optimization landscape contains multiple alternative supports that explain the data equally well, why choose only one?
The framework proposed by Satoshi Hara et al. in Enumerate Lasso Solutions for Feature Selection explicitly enumerates the "top-$k$" alternative Lasso solutions in order of their objective values. By exploring the mathematical neighborhood around the global optimum, this approach uncovers alternative feature sets that standard Lasso would arbitrarily discard, handing domain experts a complete menu of statistically equivalent models to evaluate based on scientific merit.
Distributional Proximity via E-Values
A more radical departure from traditional shrinkage is introduced by Subhabrata Majumdar et al. in Feature selection using e-values. Instead of searching through an exponential combination of sub-models or relying on iterative coefficient pruning, the authors fit only a single model—the full one.
They then evaluate potential feature subsets by comparing their sampling distributions to the full model using data depths and a novel e-value framework. By shifting the problem from coefficient optimization to distribution proximity, this mechanism completely bypasses the geometric vulnerabilities of the $\ell_1$ ball, achieving robust sparsity and lower prediction error without succumbing to correlated feature bias.