Abstract
As part of the effort to understand implicit bias of gradient descent in overparametrized models, several results have shown how the training trajectory on the overparametrized model can be understood as mirror descent on a different objective. The main result here is a characterization of this phenomenon under a notion termed commuting parametrization, which encompasses all the previous results in this setting. It is shown that gradient flow with any commuting parametrization is equivalent to continuous mirror descent with a related Legendre function. Conversely, continuous mirror descent with any Legendre function can be viewed as gradient flow with a related commuting parametrization. The latter result relies upon Nash's embedding theorem.
Introduction
In modern deep learning, models are typically overparametrized, meaning they possess significantly more parameters than data points. This redundancy creates a landscape with infinitely many global minima that can achieve zero training loss. Despite this, gradient descent (GD) consistently converges to solutions that generalize well to unseen data. This phenomenon, known as implicit bias, suggests that the choice of optimization algorithm and model architecture inherently regularizes the learning process.
Recent research has identified that changing how a model is represented—its reparametrization—can fundamentally alter this bias. For instance, expressing a weight $w$ as $u^2 - v^2$ changes the optimization trajectory. While specific instances of reparametrization have been shown to mimic mirror descent (an algorithm using non-Euclidean geometry), a unified theory explaining this connection has been absent. This paper bridges that gap by introducing the concept of commuting parametrization.
The Framework of Commuting Parametrization
The central innovation of this work is the definition of a commuting parametrization. Consider a parametrization $w = G(x)$. This parametrization is said to be "commuting" if the gradients of its individual components commute in a geometric sense—specifically, if their Lie bracket is zero.
Intuitively, this condition implies that infinitesimal steps taken in different parameter directions do not interfere with each other's progress. When this condition is met, the complex dynamics of gradient descent in the high-dimensional space of $x$ can be simplified and viewed as mirror descent in the original weight space $w$.
Theoretical Equivalence
The paper establishes a bi-directional equivalence between reparametrized gradient flow and mirror flow through two primary theorems:
1. The Forward Direction (Theorem 4.9)
The authors prove that gradient flow under any regular, commuting parametrization is mathematically equivalent to continuous-time mirror flow. In this mapping, the specific "mirror" (the Legendre function) is determined entirely by the geometry of the parametrization $G$ and the initialization $x_{init}$. Crucially, this implicit bias is loss-independent; the trajectory is constrained by the model's architecture rather than the specific objective function being minimized.
2. The Reverse Direction (Theorem 5.1)
Using Nash’s Embedding Theorem, the authors demonstrate the universality of this relationship. They prove that for any smooth mirror flow, there exists a corresponding higher-dimensional reparametrization that allows standard gradient descent to follow that exact path. This suggests that mirror descent is not merely a special case but a fundamental behavior that can be "engineered" through specific model architectures.
Case Studies and Implications
To validate the theory, the paper examines several standard settings:
- Underdetermined Linear Regression: The framework recovers and extends results showing how quadratic reparametrizations lead to sparse, entropy-like regularization.
- Matrix Factorization: The authors explain why general matrix factorization (e.g., $W = UU^\top$) does not easily map to mirror descent. In these models, the components do not commute, which clarifies why their implicit bias is significantly more complex and difficult to characterize than simpler models.
Conclusion and Future Work
This research provides a unified mathematical umbrella for understanding how architecture choices influence optimization. By establishing the equivalence between reparametrization and mirror descent, it offers a rigorous toolset for analyzing implicit regularization in overparametrized models.
However, several questions remain. The current work focuses on continuous-time "flows," whereas practical training occurs in discrete steps. Furthermore, while Nash's theorem guarantees the existence of a reparametrization for any mirror descent, finding "natural" or computationally efficient parametrizations for specific desired biases remains an open challenge. Future work may also address the "commuting gap," exploring whether the necessary conditions for this equivalence can be further relaxed.
Summary updated