In the search for sparse solutions—where we want to represent data using the fewest possible components—the $L_1$ norm has long been the industry standard. It is convex, easy to optimize, and serves as a reliable proxy for the "true" sparsity measure, the $L_0$ norm (which simply counts non-zero entries).
However, $L_1$ minimization has a fundamental flaw: it is scale-dependent. If you double the magnitude of your signal, you must manually retune your regularization hyperparameters. Furthermore, $L_1$ suffers from "shrinkage bias," where it suppresses the magnitude of the very features it is trying to preserve.
A growing body of research suggests a more robust alternative: using the ratio of norms, such as $L_1/L_2$, as a scale-invariant proxy for sparsity. This approach, often referred to as the Hoyer regularizer or quotient regularization, offers a parameter-free way to recover sparse signals and low-rank structures without the systematic biases of convex methods.
The Intuition of Peakiness
To understand why a ratio of norms promotes sparsity, consider the geometry of the $L_1$ and $L_2$ norms. The $L_1$ norm ($\sum |x_i|$) measures the total "mass" of a vector, while the $L_2$ norm ($\sqrt{\sum x_i^2}$) measures its energy.
As Ernie Esser and Jack Xin describe in A Method for Finding Structured Sparse Solutions to Non-negative Least Squares Problems with Applications, the ratio $L_1/L_2$ effectively measures the "peakiness" of a distribution. A vector with only one non-zero entry (highly sparse) has the lowest possible ratio, while a "flat" vector where every entry is equal has the highest.
Crucially, this ratio is scale-invariant. If you multiply a signal by a constant, both the numerator and denominator scale equally, leaving the ratio unchanged. This property allows the regularizer to focus purely on the structure of the signal rather than its absolute magnitude.
Eliminating the Shrinkage Bias
Traditional $L_1$ regularization penalizes the absolute magnitude of coefficients. This creates a "chicken-and-egg" problem in fields like compressed sensing: to recover a signal, you need to know its sparsity, but you cannot know the sparsity until you have recovered the signal. Miles Lopes argues in Estimating Unknown Sparsity in Compressed Sensing that numerical sparsity (the ratio of squared norms) provides a much more stable measure of a signal's effective support.
Because the $L_1/L_2$ ratio does not penalize the overall magnitude, it avoids the "contrast loss" common in image processing. Chao Wang and Yifei Lou demonstrate in Minimizing L1 over L2 norms on the gradient that this approach preserves the intensity of edges and features in MRI reconstructions and image denoising, whereas standard Total Variation (based on $L_1$) tends to over-smooth and dim the resulting image.
Applications Across Domains
The benefits of scale invariance have led to the adoption of norm ratios across diverse research areas:
- Deep Learning Compression: In Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification, Huanrui Yang and colleagues use the Hoyer regularizer to prune the singular values of weight matrices. This allows for aggressive model compression (up to 6.42x speedups) by pushing insignificant singular values to zero while preserving the energy of the dominant ones.
- Mechanistic Interpretability: When analyzing Large Language Models, Mingyang Deng and Lucas Tao use scale-invariant metrics to measure Feature Sparsity. This allows researchers to compare sparsity across different layers and architectures even when the activation scales vary wildly.
- Genomics and Imaging: In sparse parallel independent component analysis (spICA), Kuaikuai Duan uses the Hoyer projection to identify genetic variants linked to brain structure. Scale invariance is critical here because ICA produces components with arbitrary scales, making standard $L_1$ constraints inconsistent.
Overcoming the Non-Convex Nightmare
The primary challenge of using ratios like $L_1/L_2$ is that they are non-convex, making them significantly harder to optimize than the $L_1$ norm. However, recent algorithmic advances have made these models practical.
Yaghoub Rahimi and Chao Wang have developed efficient solvers based on the Alternating Direction Method of Multipliers (ADMM) and gradient flow. In Accelerated Schemes for the $L_1/L_2$ Minimization, they show that these methods are particularly effective in "coherent" settings—scenarios where the measurement columns are highly correlated and standard $L_1$ methods typically fail.
By treating sparsity as a ratio rather than a sum, these researchers have provided a framework that is not only more mathematically robust but also more "set-and-forget" for practitioners working with real-world, variable-scale data.