In the search for sparse signals—where we assume only a few variables out of many are truly influential—the $\ell_1$ norm (or Lasso) has long been the industry standard. It is computationally efficient and mathematically elegant. However, it harbors a fundamental flaw: it is a biased estimator. While it is excellent at identifying which components are active (the support), it systematically underestimates how strong those components are.
This phenomenon, known as shrinkage bias or amplitude bias, occurs because the mathematical tools we use to promote sparsity also inadvertently penalize the magnitude of the signal we are trying to recover.
The Sparsity-Fidelity Tradeoff
To understand why this happens, consider the standard optimization objective for sparse recovery:
$$\min_{x} \frac{1}{2} \|Ax - b\|_2^2 + \lambda \|x\|_1$$
The first term ensures "fidelity" (the model should fit the data), while the second term, the $\ell_1$ penalty, promotes "sparsity." As Stanley Osher and Wotao Yin explain in Sparse Recovery via $\ell_1$ and $L_1$ Optimization, the $\ell_1$ norm is a convex relaxation of the ideal $\ell_0$ "norm," which simply counts non-zero elements.
The problem is that the $\ell_1$ norm is not just a counter; it is a function of the signal's magnitude. Because the penalty increases as the absolute value of the coefficients increases, the optimizer is incentivized to "shrink" every non-zero coefficient toward zero to minimize the total cost. This results in a systematic downward bias. As Alexandre Belloni notes in High Dimensional Sparse Econometric Models, this bias can distort the estimated effects of even the most significant predictors.
Why Soft-Thresholding Shrinks
The mechanism of this bias is most visible in the soft-thresholding operator, the core engine of many iterative solvers like ISTA. As Jialin Liu describes in Theoretical Linear Convergence of Unfolded ISTA, soft-thresholding doesn't just zero out small noise; it subtracts a constant value from every remaining coefficient.
Imagine a signal with a true value of 10 and a noise floor of 1. A hard-thresholding approach might keep the 10 as it is. Soft-thresholding, however, might shift that 10 down to 9. In image processing, this manifest as a loss of contrast or "staircasing" artifacts, where smooth gradients are flattened into blocks because the penalty is constantly pulling values toward the background (Deledalle et al.).
Strategies for Unbiased Recovery
Researchers have developed several distinct strategies to "undo" this shrinkage while maintaining the benefits of sparse selection.
1. Post-Selection Refitting
The most intuitive fix is a two-stage process: use a sparse regularizer to find the "support" (the non-zero indices), then throw the penalty away and run a standard Least Squares regression on only those indices. This technique, often called Post-Lasso, is shown by Belloni to match the convergence rates of Lasso while significantly reducing bias (Post-L1-Penalized Estimators). Similar "refitting" logic is applied in BoATS to recover true signal strengths in neuroscience data.
2. Iterative Reweighting
If the $\ell_1$ penalty's flaw is that it penalizes large coefficients too much, why not change the weights? Emmanuel Candes and Michael B. Wakin introduced Reweighted L1 Minimization (Enhancing Sparsity by Reweighted L1 Minimization), which solves a sequence of weighted problems. By setting weights inversely proportional to the magnitude of the signal from the previous step, the algorithm "learns" to stop penalizing large, legitimate components, effectively mimicking the unbiased $\ell_0$ norm.
3. Non-convex Penalties
Another approach is to move away from convex functions entirely. Non-convex penalties, such as the $\ell_p$ quasi-norm (where $p < 1$), have a "flatter" tail, meaning they penalize large values much less than the $\ell_1$ norm does. Daniel Strohmeier and Alexandre Gramfort demonstrate that using an $\ell_{0.5}$ penalty in M/EEG imaging significantly reduces amplitude bias and eliminates "ghost" sources that appear when the model tries to compensate for over-shrunk signals (The Iterative Reweighted Mixed-Norm Estimate).
4. Bregman Iterative Methods
A more dynamic solution involves the Bregman Inverse Scale Space. Instead of solving a static optimization problem with a fixed $\lambda$, this method treats the recovery as a path. Osher et al. proved that this path contains a point that is both sign-consistent and completely unbiased—effectively reaching the "oracle estimator" through the simple mechanism of early stopping (Sparse recovery via differential inclusions).
Impact Across Domains
The quest for unbiased recovery is not just a theoretical exercise; it is critical for physical accuracy. In solar physics, Anna Jungbluth and colleagues found that standard losses bias models toward the zero-valued background, failing to reconstruct rare but vital high-magnitude magnetic features (Single-Frame Super-Resolution of Solar Magnetograms). Similarly, in reinforcement learning, Botao Hao and Csaba Szepesvári note that shrinkage bias can distort value function estimates, leading to sub-optimal policies even when the correct features are identified (Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient).