Introduction
In modern machine learning, models are frequently trained with far more parameters than data points. In these overparametrized regimes, there are infinitely many solutions that can achieve zero training loss. Yet, gradient descent (GD) consistently converges to solutions that generalize well to unseen data. This phenomenon, known as implicit bias, suggests that GD inherently favors certain types of solutions over others.
Researchers have long suspected that GD on reparametrized models (such as neural networks) behaves as if it is minimizing a hidden penalty or regularizer. Previous studies found specific examples—like in matrix factorization or "diagonal" linear networks—where GD behaves like mirror descent, an optimization algorithm that moves through parameter space according to a specific geometry defined by a potential function. However, a general theory was missing: When exactly does reparametrization lead to mirror descent, and can any mirror descent be represented as a reparametrized gradient flow?
This paper provides a unified theoretical framework to answer these questions, proving a mathematical equivalence between gradient flow on reparametrized models and mirror descent.
The Geometry of Reparametrization
The authors bridge the theoretical gap by focusing on the continuous-time limit of gradient descent, known as gradient flow. Their central technical contribution is the formalization of a commuting parametrization.
Suppose we have a model with parameters $w$ that we reparametrize as $w = G(x)$. This parametrization is considered "commuting" if the gradients of the individual components of $G$ commute with each other in a geometric sense—specifically, their Lie bracket is zero.
Using this definition, the authors establish a profound two-way equivalence:
- Sufficiency (Gradient Flow to Mirror Descent): If a parametrization is commuting and "regular," the resulting gradient flow in the $x$-space is mathematically equivalent to a mirror flow in the $w$-space. The authors provide a concrete recipe to construct the corresponding "mirror map" (Legendre function) directly from the mapping $G$.
- Universality (Mirror Descent to Gradient Flow): To prove the reverse—that any mirror descent can be simulated by some reparametrization—the authors utilize differential geometry. Leveraging Nash’s embedding theorem, they demonstrate that the geometry induced by any mirror descent can be realized by embedding the parameters into a higher-dimensional Euclidean space.
Key Insights and Theoretical Results
This framework encompasses many previously disconnected results in the literature, placing them under a single theoretical roof.
- The "Commuting" Boundary: The commuting property serves as the boundary between "simple" implicit bias (mirror descent) and more complex optimization dynamics. If the components of a reparametrization do not geometrically interfere with each other's directions, the optimization yields the clean, predictable behavior of mirror descent.
- Architecture as Geometry: The paper establishes that changing the architecture of a model via reparametrization is fundamentally equivalent to changing the geometry of the optimization algorithm via mirror descent.
- Rigorous Convergence: Unlike prior works that merely assumed parameters would eventually converge to a solution, this paper provides formal proofs for parameter convergence in these commuting settings.
- Linear Regression Recovery: The authors successfully recover known implicit bias results for quadratically reparametrized linear models (e.g., $w = u^2 - v^2$, which induces sparsity-favoring $\ell_1$-like regularization), demonstrating that their general theory perfectly matches existing specialized cases.
Limitations and Future Directions
While the theoretical equivalence is powerful, the authors note several limitations that pave the way for future research:
- Non-Commuting Models: Many standard deep learning architectures, such as multi-layer perceptrons or non-diagonal matrix factorization, are non-commuting. For these models, the training trajectory cannot be simplified into a standard mirror flow. Understanding the implicit bias of these complex, non-commuting models remains a major open challenge.
- Discrete vs. Continuous Time: The theoretical results apply to continuous gradient flow. Real-world training relies on discrete gradient descent steps, which can introduce "drift" that the continuous theory does not fully capture.
- Relaxing the Commuting Condition: The authors conjecture that the strict "commuting" condition might be slightly stronger than necessary, suggesting that a weaker geometric condition might still suffice to trigger mirror descent behavior.
Summary updated