When estimating parameters in high-dimensional settings where the number of features $p$ exceeds the number of observations $n$, regularization is required to prevent overfitting and enforce sparsity. The most common tool for this is $L_1$ regularization (the Lasso). However, $L_1$ regularization introduces a fundamental trade-off: the very mechanism that enforces sparsity also introduces a systematic shrinkage bias.
To understand why, consider the optimization objective. The $L_1$ penalty adds a cost proportional to the absolute value of the coefficients. This acts like a constant force pulling every coefficient toward zero. While this successfully zeroes out noisy, irrelevant features, it also truncates the magnitude of the truly significant features. As noted by Javanmard et al. in Confidence Intervals and Hypothesis Testing for High-Dimensional Regression, this shrinkage makes it incredibly difficult to characterize the estimator's probability distribution, preventing the use of classical statistical tools like p-values or confidence intervals.
Why Shrinkage Breaks Statistical Inference
In classical statistics, an unbiased estimator is centered directly on the true parameter value, allowing us to construct symmetric confidence intervals using standard normal distributions. Shrinkage bias destroys this symmetry. Because the regularized estimator systematically underestimates the true parameter values, its sampling distribution is shifted toward zero.
If you attempt to compute a p-value using a standard Lasso estimate, your test statistic will be severely distorted. This issue spans across various domains, from mapping brain connectivity networks Gretton et al. to reconstructing sparse MRI images Hoppe et al.. Without correcting for this bias, we cannot distinguish between a feature that is truly zero and one that has simply been aggressively shrunk by the penalty.
Analytical De-biasing and Desparsification
The most direct way to restore our ability to perform inference is to mathematically "undo" the shrinkage after the initial regularized optimization. This approach is known as de-biasing or desparsifying.
The intuition behind the de-biased Lasso, developed by Javanmard et al. in Confidence Intervals and Hypothesis Testing for High-Dimensional Statistical Models, is to take the biased sparse estimator and add a correction term based on the residuals. Specifically, the method computes a surrogate inverse of the data's covariance matrix. By multiplying the residuals of the initial model by this surrogate inverse, the algorithm projects the omitted signal back onto the coefficients, effectively pulling them back up to their true scale.
This correction step yields an estimator that is no longer sparse, but is asymptotically normal and centered at the true parameter value. This allows researchers to construct valid confidence intervals and calculate exact p-values, a technique applied to complex spatio-temporal imaging by Salmon et al. and to multi-layered biological networks by Majumdar et al.. When computing normalizing constants is computationally intractable, alternative scoring rules like the Hyvärinen scoring rule can be leveraged to achieve a similar $\sqrt{n}$-consistent, de-biased estimator Yu et al..
Nonconvex Penalties and Multi-Step Refinement
Instead of correcting the bias after estimation, an alternative strategy is to modify the regularization penalty itself so that it stops shrinking large coefficients. The $L_1$ penalty increases linearly with the size of the coefficient, meaning large signals are penalized just as heavily as small ones.
Nonconvex penalties, such as the Minimax Concave Penalty (MCP), solve this by altering the penalty's geometry. As the magnitude of a coefficient grows past a certain threshold, the derivative of the penalty drops to zero. In other words, once a feature proves itself to be highly significant, the model stops penalizing it. As Zhang demonstrates in A Concave Conjugate Approach for Nonconvex Penalized Regression with the MCP Penalty, this approach eliminates the estimation bias for large coefficients while maintaining clean sparsity for the rest of the model. A similar mathematical foundation using Laplace exponents of Lévy subordinators is explored by Zhang et al. to achieve cleaner sparsity without the baggage of convex relaxation bias.
Another intuitive way to eliminate this bias is through multi-step thresholding procedures, as explored in Thresholding Procedures for High Dimensional Variable Selection and Statistical Estimation. Here, a regularized model like the Lasso is used purely for variable selection to isolate a sparse subset of active features. Once the active features are identified, the penalty is discarded entirely, and standard Ordinary Least Squares (OLS) is run on just that subset to get unbiased parameter estimates.
Contamination in Causal Inference
Shrinkage bias becomes particularly dangerous when machine learning models are used to estimate causal effects. In a typical causal inference setup, we want to estimate the effect of a treatment on an outcome while controlling for high-dimensional confounding variables (nuisance functions).
If we use a regularized model (like a random forest or Lasso) to estimate these nuisance functions, the regularization bias will leak directly into our target causal estimate. As Bach et al. explain in the context of DoubleML, this bias does not disappear as the sample size grows, rendering standard confidence intervals invalid.
To circumvent this, Double Machine Learning (DML) uses an orthogonalization strategy. By choosing a score function that is insensitive to small errors in the nuisance parameters, DML isolates the target parameter from the regularization bias of the underlying machine learning models. This allows researchers to use highly flexible estimators while still obtaining valid, unbiased causal inferences Tian et al.. Furthermore, Chen et al. show that if the underlying estimators are stable, this de-biasing can even be achieved without the data-wasting requirement of sample-splitting.
Decoupling Magnitude from Detection in Deep Learning
The challenges of shrinkage bias are not confined to classical linear models; they have recently emerged as a major bottleneck in deep learning mechanistic interpretability. To understand the internal representations of large language models, researchers use Sparse Autoencoders (SAEs) trained with an $L_1$ penalty to decompose dense activations into a sparse set of human-interpretable features.
However, just like in linear regression, the $L_1$ penalty forces the autoencoder to systematically underestimate feature magnitudes to satisfy the sparsity constraint. This feature shrinkage degrades the reconstruction fidelity of the network.
To solve this, Rajamanoharan et al. introduced architectural modifications that decouple feature detection from magnitude estimation. In Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders, the authors propose a gated mechanism where one path determines whether a feature is active, while an unpenalized path estimates its magnitude. By completely eliminating shrinkage bias, these architectures achieve a massive Pareto improvement, delivering identical reconstruction fidelity with a fraction of the active features. A similar leap in fidelity is achieved via JumpReLU activations Rajamanoharan et al., which replace the smooth, shrinking ramp of a standard ReLU with a sharp, discontinuous threshold.