When building predictive models from high-dimensional datasets—where the number of measured variables ($p$) vastly exceeds the number of available observations ($n$)—traditional statistical approaches break down. In fields like genomics, radiomics, and high-density signal processing, models face the "curse of dimensionality." If a model attempts to assign a weight to every single available feature, it inevitably fits the random noise within the training data rather than the underlying signal, leading to poor generalization on unseen data.
To overcome this, researchers rely on sparse feature selection. The goal is to automatically identify and retain only the most informative predictors while discarding the rest by setting their coefficients exactly to zero. The primary mechanism driving this behavior is $L_1$ regularization, commonly known as the Least Absolute Shrinkage and Selection Operator (LASSO).
The Geometry of Sparsity: Why L1 Regularization Works
To understand why $L_1$ regularization produces sparse models while other regularization methods do not, it helps to look at the geometry of the optimization problem.
When training a standard linear or logistic regression model, we minimize a loss function (such as mean squared error or cross-entropy) that measures how well the model fits the data. In a regularized regression model, we add a penalty term to this loss function to constrain the magnitude of the model's coefficients ($\beta$).
Consider the difference between $L_2$ regularization (Ridge regression) and $L_1$ regularization (LASSO):
- $L_2$ Regularization (Ridge): The penalty is proportional to the sum of the squared coefficients ($\sum \beta_i^2$). Geometrically, in a two-dimensional parameter space, this constraint defines a smooth, spherical budget region (a circle). As the contours of the loss function expand from the unconstrained optimal fit, they are highly likely to intersect the smooth boundary of the $L_2$ circle at a point where both coefficients are non-zero. Consequently, $L_2$ regularization shrinks weights close to zero but rarely sets them exactly to zero.
- $L_1$ Regularization (LASSO): The penalty is proportional to the sum of the absolute values of the coefficients ($\sum |\beta_i|$). Geometrically, this constraint defines a diamond-shaped budget region with sharp corners situated precisely on the coordinate axes (where $\beta_1 = 0$ or $\beta_2 = 0$). When the expanding loss contours meet this diamond, they are geometrically biased to hit one of these sharp corners first. Hitting a corner means that one or more coefficients are driven completely to zero.
By penalizing the absolute size of the regression coefficients, $L_1$ regularization acts as a simultaneous feature selector and regularizer. It filters out redundant or uninformative variables, mitigates multicollinearity, and yields a parsimonious, interpretable model.
Distilling High-Dimensional Medical Imaging
The power of sparsity-driven feature selection is highly evident in radiomics, where thousands of quantitative features—such as shape, texture, and intensity metrics—are extracted from medical scans like CT, MRI, or ultrasound.
Without regularization, a machine learning model would easily overfit these imaging datasets. For instance, Tang et al. applied LASSO to reduce thousands of MRI-extracted features down to a robust, low-dimensional signature capable of predicting the malignancy grade of pancreatic neoplasms (Radiomic nomogram based on MRI to predict grade of branching type intraductal papillary mucinous neoplasms of the pancreas: a multicenter study). Similarly, Zhang et al. utilized LASSO to eliminate redundant features from optic chiasm MRIs, successfully predicting visual recovery after surgery while preventing model overfitting (Machine Learning-Based Radiomics of the Optic Chiasm Predict Visual Outcome Following Pituitary Adenoma Surgery).
This approach scales across various imaging modalities and clinical objectives:
- Ultrasound: Ben et al. used sparsity-driven selection to isolate key microflow imaging features to predict thyroid cancer metastasis (Multimodal ultrasound radiomics containing microflow images for the prediction of central lymph node metastasis in papillary thyroid carcinoma), while Zhang et al. leveraged LASSO to ensure their deep-learning-enhanced ultrasound model remained generalizable across different hospital populations (Deep learning-based super-resolution US radiomics to differentiate testicular seminoma and non-seminoma: an international multicenter study).
- Computed Tomography (CT): Chen et al. used LASSO to identify a minimal subset of predictive variables for thyroid carcinoma extension (Computed Tomography Radiomic Nomogram for Preoperative Prediction of Extrathyroidal Extension in Papillary Thyroid Carcinoma), and Cao et al. integrated automated segmentation with sparse feature selection to noninvasively classify gastric cancers (Preoperative prediction of the Lauren classification in gastric cancer using automated nnU-Net and radiomics: a multicenter study).
- Tumor Invasiveness: Guofo Ma et al. demonstrated that LASSO could successfully isolate a handful of highly predictive variables from a massive feature pool to determine craniopharyngioma invasiveness before surgery (Non-Invasive Radiomics Approach Predict Invasiveness of Adamantinomatous Craniopharyngioma Before Surgery), and the single-author study by Zhao showed that regularized selection filters out background noise to accurately differentiate rare esophageal sarcomatoid carcinomas from more common squamous cell carcinomas (Investigating esophageal sarcomatoid carcinoma and its comparison with esophageal squamous cell carcinoma on clinicopathological characteristics, prognosis, and radiomics features: a retrospective study).
Sifting Through Genomic and Biomarker Data
Genomics presents another classic $p \gg n$ challenge, where researchers measure expressions of thousands of genes, microRNAs, or methylation sites from a relatively small cohort of patients.
To find a true biological signal, models must be highly selective. Li et al. used Lasso regression to narrow down a vast field of candidate genes to identify EPYC as a critical prognostic biomarker for pancreatic cancer (EPYC functions as a novel prognostic biomarker for pancreatic cancer). When dealing with the extreme dimensionality of DNA methylation, Haftorn et al. paired LASSO with a stability selection framework, enabling them to distill thousands of potential sites down to just five highly consistent, predictive CpGs for determining gestational age (Stability selection enhances feature selection and enables accurate prediction of gestational age using only five DNA methylation sites).
L1-penalized regression models serve as a foundational tool across diverse molecular and genetic discovery pipelines:
- Diagnostic Signatures: Poore et al. used sparse logistic regression to compress high-dimensional host microRNA responses into a compact diagnostic signature that differentiates viral from bacterial infections (A miRNA Host Response Signature Accurately Discriminates Acute Respiratory Infection Etiologies).
- Prognostic Modeling: Cao et al. constructed a parsimonious breast cancer survival model by isolating five key anoikis-related genes (Prognostic analyses of genes associated with anoikis in breast cancer), while Li et al. used LASSO to confirm the prognostic value of CD69 in hepatocellular carcinoma (CD69 serves as a potential diagnostic and prognostic biomarker for hepatocellular carcinoma).
- Pathway Analysis: Liang et al. applied LASSO logistic regression to filter differentially expressed genes within the TGF-β signaling pathway to uncover biomarkers for acute myocardial infarction (Analysis of immunoinfiltration and EndoMT based on TGF-β signaling pathway-related genes in acute myocardial infarction).
- Tumor Classification: Soh et al. demonstrated that $L_1$-regularized logistic regression can simultaneously classify and select features to accurately predict cancer types from somatic mutations and copy number alterations (Predicting cancer type from tumour DNA signatures).
Designing Interpretable Clinical Nomograms
In clinical practice, a predictive model is useless if it requires a doctor to manually input dozens of variables during a busy shift. Models must be simple, interpretable, and visual—often formatted as a "nomogram." LASSO is uniquely suited for this because its mathematical formulation naturally discards weak or redundant clinical indicators.
For example, Si et al. used LASSO to shrink less important clinical and demographic variables to zero, leaving a clean, visual nomogram for preoperative prediction of lymph node involvement in thyroid microcarcinoma (Development and validation of a nomogram for preoperative prediction of cervical lymph node involvement in thyroid microcarcinoma). Similarly, Li et al. demonstrated that a LASSO-based model achieved superior generalizability over traditional logistic regression by actively preventing overfitting when predicting neurocognitive disorders in ICU patients (Development and validation of prediction models for neurocognitive disorders in adult patients admitted to the ICU with sleep disturbance).
This strategy of building parsimonious clinical tools is widely replicated:
- Chen et al. used LASSO to reduce a high-dimensional set of radiological semantic and clinical features into a compact, highly accurate diagnostic model for COVID-19 (A diagnostic model for coronavirus disease 2019 (COVID-19) based on radiological semantic and clinical features: a multi-center study).
- Gao et al. applied Lasso logistic regression to isolate key clinical imaging features to predict lymph node metastasis in patients with mixed ground-glass nodules ([Prediction of Lymph Node Metastasis of Mixed Ground-glass Nodules Based on Clinical Imaging Information].).
- Wang et al. combined PET/CT radiomics with routine clinical metrics, using LASSO to drive irrelevant features to zero and create a streamlined risk stratification tool for prostate cancer (Evaluation of a radiomics nomogram derived from Fluoride-18 PSMA-1007 PET/CT for risk stratification in newly diagnosed prostate cancer).
Applications Beyond Medicine
The utility of sparse feature selection extends far beyond clinical diagnostics. Any domain that generates high-dimensional data or requires interpretable, low-dimensional representations benefits from the geometric properties of $L_1$ regularization.
Biomechanics and Signal Processing
In mobile health and biomechanics, De Brabandere et al. applied $L_1$-regularized linear regression to identify the most relevant predictors from high-dimensional mobile sensor data, allowing them to estimate knee and hip joint loading using only a single phone-embedded IMU (A Machine Learning Approach to Estimate Hip and Knee Joint Loading Using a Mobile Phone-Embedded IMU). In neurological signal processing, Chaturvedi et al. used LASSO-penalized regression to distill 79 complex spectral EEG measures down to just six key features, creating a compact "brain fingerprint" for Parkinson's disease (Quantitative EEG (QEEG) Measures Differentiate Parkinson's Disease (PD) Patients from Healthy Controls (HC)).
Speech and Behavioral Science
In computer-assisted language learning, Dong et al. used Lasso regression to isolate the specific acoustic features and speech recognition metrics that best predict human evaluations of non-native pronunciation progress (Evaluating Progress of CALL System Users on Accentedness and Comprehensibility: An Acoustic and ASR-Based Approach). Similarly, Tawfik et al. used Lasso as a regularized feature selection method to sift through a large pool of survey covariates, identifying technology frustration as a leading independent predictor of healthcare worker burnout (Frustration With Technology and its Relation to Emotional Exhaustion Among Health Care Workers: Cross-sectional Observational Study).
Causal Inference and Economics
In econometrics, selecting the wrong control variables can bias causal estimates. To solve this, Hansen et al. introduced a "double-selection" mechanism that uses LASSO to independently select which control variables and instrumental variables are truly relevant, ensuring valid statistical inference even in high-dimensional settings (Post-Selection and Post-Regularization Inference in Linear Models with Many Controls and Instruments).
Urban Planning and Real Estate
In spatial analysis, Durán-Rodas et al. utilized LASSO regression to isolate the specific demographic and social groups most predictive of bike-sharing infrastructure demand (How fair is the allocation of bike-sharing infrastructure? Framework for a qualitative and quantitative spatial fairness assessment). In real estate modeling, the single-author paper by Wang demonstrated how Lasso regression can be used to identify and filter out highly correlated independent variables to reduce noise and improve price prediction models (Predicting Airbnb Listing Price with Different models).
Network Inference and Subsampling
The principles of $L_1$ optimization also underpin structural network discovery. The authors of A linear programming approach for estimating the structure of a sparse linear genetic network from transcript profiling data framed genetic network inference as a series of sparse linear regressions, using an $L_1$ norm penalty to drive non-existent regulatory links to zero and map biologically plausible pathways. Finally, Li et al. explored how the mathematical foundations of feature selection can be inverted to optimize data subsampling, leveraging algorithmic connections to select the most informative data points for high-dimensional ridge regression (Optimal subsampling for high-dimensional ridge regression).