In modern machine learning, models are frequently overparameterized, possessing far more parameters than training data points. Despite the existence of infinite solutions that perfectly fit the data, standard optimization algorithms like gradient descent consistently find solutions that generalize well. This phenomenon, known as implicit bias, suggests that the optimization process itself acts as a regularizer.
This research provides a mathematical "cookbook" for engineering this implicit bias. By strategically choosing how a model is parameterized, researchers can induce specific types of regularization—such as $\ell_p$-norm or sparsity-promoting penalties—without ever adding an explicit penalty term to the loss function.
The Framework: Reparametrized Gradient Flow
The core idea involves optimizing a parameter $z$ such that the actual model weights $w$ are defined by a reparametrization function: $w = \rho_{rp}(z)$. Instead of performing gradient descent directly on $w$, the flow is directed through $z$.
Separation of Concerns
A key insight of this work is the "separation of concerns" between the model's architecture and its optimization. The authors demonstrate that:
- The Reparametrization ($\rho_{rp}$) determines the type of regularization (the "what").
- The Loss Function ($L$) and Link Function ($\rho_{link}$) determine the convergence behavior and speed (the "how fast").
By using Bregman divergences, the authors prove that the solution found by gradient flow is the one that minimizes a specific potential function $F$ (derived from the reparametrization) while satisfying the data constraints. This provides a systematic way to translate a model's architecture into its underlying regularization strategy.
Inducing Specific Regularization Behaviors
$\ell_p$ Norms and Sparsity
By setting the reparametrization to a specific power function, researchers can interpolate between different types of regularization. For instance, a reparametrization can be tuned to mimic $\ell_1$ (promoting sparsity) or $\ell_2$ (standard least squares) behavior. This allows for fine-tuned control over the "sharpness" of the resulting solution.
As shown in the figure above, the geometry of the $\ell_p$-ball changes based on the parameter $p$. When $p=1.2$, the iterates are biased toward sparser solutions (closer to the axes), whereas $p=1.8$ results in a more rounded ball and denser solutions.
Trigonometric Reparametrization and Huber-like Bias
Another powerful technique involves using trigonometric functions like sinh(z) or tanh(z). These reparametrizations induce a bias that behaves similarly to the Huber loss.
As illustrated in Figure 4, the induced potential function $g_{sinh}$ closely tracks the Huber loss. This means the model acts like it has an $\ell_1$ penalty for large parameter values (promoting sparsity) but remains smooth and $\ell_2$-like near zero, avoiding the optimization difficulties often associated with non-smooth $\ell_1$ penalties.
Convergence and Initialization
The "strength" of this induced regularization is heavily dependent on the initialization scale. Small initializations lead to stronger, more principled implicit biases that more closely approximate true norm-minimizers.
Mathematically, the authors provide rigorous proofs for convergence:
- Theorem 2.4 (Implicit Bias): Provides a general formula to calculate the potential function $H$ for any given smooth, invertible reparametrization.
- Convergence Rates: The objective value is proven to decay at a rate of $O(1/t)$ for general convex losses. If the loss satisfies the Polyak-Lojasiewicz (PL) condition, the system achieves linear convergence ($O(e^{-ct})$).
Conclusion and Future Directions
This framework moves beyond specific case studies (like diagonal linear networks) to provide a general theory of how reparametrization shapes the optimization landscape. While currently focused on entry-wise reparametrizations in Generalized Linear Models (GLMs), this work lays the foundation for understanding implicit bias in deeper, more complex architectures. Future research will need to address how these principles scale to non-linear layers and convolutional structures where parameters interact more intricately.
Summary updated