Why do deep neural networks generalize well even when they have far more parameters than data points? The classical machine learning paradigm suggests that overparameterized models should overfit perfectly to noise. The standard resolution to this paradox points to the implicit bias of optimization algorithms: standard gradient descent (GD) naturally gravitates toward minimum-norm solutions that happen to generalize well.
However, this explanation treats the network architecture as a passive container for parameters. In reality, the architecture is an active participant. The specific structural design of a network—its layers, connectivity, and weight mappings—acts as a powerful implicit regularizer. By acting as a coordinate transformation on the parameter space, the architecture warps the optimization landscape, guiding standard gradient descent toward highly structured, simple, and physically plausible solutions without requiring any explicit penalty terms.
Reparameterization as a Coordinate Transformation
To build intuition for why architecture introduces bias, consider a simple linear model $y = w^T x$. If we optimize the weight vector $w$ directly using gradient descent, the algorithm minimizes the standard Euclidean ($\ell_2$) norm of $w$.
Now, consider a trivial architectural change: we reparameterize each weight as the square of an underlying parameter, $w_i = u_i^2$, and run gradient descent on $u$ instead. As shown by Ehsan Amid et al. in Winnowing with Gradient Descent, this simple change completely alters the optimization dynamics. Instead of minimizing the $\ell_2$ norm, gradient descent on $u$ implicitly minimizes the $\ell_1$ norm of $w$, driving the model toward sparse solutions.
Why does this happen? The architecture acts as a coordinate transformation. When gradient descent takes a step in the $u$-space, the chain rule dictates how that step translates to the actual weight $w$:
$$\frac{dw_i}{dt} = -\frac{\partial L}{\partial u_i} \frac{\partial w_i}{\partial u_i} = -2u_i \frac{\partial L}{\partial u_i} = -4w_i \frac{\partial L}{\partial w_i}$$
Because the update step size is scaled by $w_i$, weights that are already large change rapidly, while weights near zero stall completely. This multiplicative update behavior is identical to classical sparsity-inducing algorithms like Winnow or Exponentiated Gradient. As Chulhee Yun et al. demonstrate in A unifying view on implicit bias in training linear neural networks, the network architecture functions as a geometric map that shapes the optimization landscape, determining the specific implicit bias of the resulting solution.
The Equivalence to Mirror Descent
This phenomenon is rooted in a deep mathematical duality. Zhiyuan Li et al. prove in Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent that gradient flow on a reparameterized model is mathematically equivalent to Mirror Descent on the original model.
Mirror descent is an optimization framework that discards standard Euclidean distance in favor of a geometry defined by a strictly convex potential function. By choosing a specific network structure, we are implicitly choosing a specific mirror potential. For instance, Ehsan Amid et al. show in Reparameterizing Mirror Descent as Gradient Descent that multi-layer configurations can be shown to be equivalent to single-layer models trained with sparsity-inducing mirror descent.
As Gal Vardi notes in On the Implicit Bias in Deep-Learning Algorithms, structural choices like diagonal layers or matrix factorization impose implicit biases toward properties like sparsity or low-rank solutions by constraining the optimization path. This mathematical link allows us to analytically derive the exact regularizer minimized by an architecture simply from its mapping function.
How Structural Components Map to Regularizers
When we move from simple reparameterizations to standard deep learning components, the architecture continues to dictate the form of regularization:
- Convolutions: Even when a linear convolutional network and a fully connected network can represent the exact same functional space, their optimization paths diverge. Jason D. Lee et al. show in Implicit Bias of Gradient Descent on Linear Convolutional Networks that convolutional structures bias gradient descent toward sparsity in the frequency domain. This geometric path is further analyzed by El Mehdi Achour et al. in The Riemannian Geometry associated to Gradient Flows of Linear Convolutional Networks, showing how weight sharing and sparsity naturally enforce stable gradient flows.
- Pooling and Group Sparsity: Tolga Ergen et al. in Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time establish that specific architectural components are mathematically equivalent to particular convex regularizers in a dual optimization problem. For example, average pooling acts precisely as a group sparsity regularizer, while circular convolutions induce spectral sparsity. Similarly, Jiangyuan Li et al. demonstrate in Implicit Regularization for Group Sparsity that diagonally grouped linear networks naturally force group-sparse solutions without explicit penalties.
- Coordinate-MLPs: The architecture's dominance over the optimizer is highlighted by Sameera Ramasinghe in On the Frequency-bias of Coordinate-MLPs, who demonstrates that the implicit regularization of a model is primarily determined by its architecture rather than the optimization process alone; networks designed for high-frequency capture can completely bypass the typical preference for smooth functions.
The Simplicity Bias of Modern Architectures
This structural regularization manifests as a pervasive "simplicity bias" across complex, modern architectures. As argued by Micah Goldblum et al. in Position: The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning, natural data possesses low Kolmogorov complexity, and modern neural networks generalize because their structural choices inherently prefer these simple patterns.
This is particularly evident in Transformers. Bhavya Vasudeva et al. in Transformers Learn Low Sensitivity Functions: Investigations and Implications discover that the Transformer architecture is uniquely biased to learn "low sensitivity" functions—where outputs remain stable despite slight perturbations to individual input tokens. This architectural preference explains why they find flatter minima and generalize robustly.
Furthermore, this structural bias is so potent that it exists even before training begins. Ziqian Zhong et al. demonstrate in Algorithmic Capabilities of Random Transformers that randomly initialized transformers with frozen internal weights possess an innate inductive bias that implements complex algorithmic computations in low-dimensional subspaces, requiring only the interface layers to be trained.
Ultimately, understanding neural networks requires moving past the view that optimization and architecture are separate concerns. The architecture is the optimization geometry. By carefully designing structural constraints, weight mappings, and layer configurations, we do not just alter what a network can represent—we dictate what it prefers to learn.
Key Foundational Work
Implicit Bias of Gradient Descent on Linear Convolutional Networks (2018) — contributed 3 concept elements to this direction.
Reparameterizing Mirror Descent as Gradient Descent (2020) — contributed 3 concept elements to this direction.
The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings (2018) — contributed 1 concept elements to this direction.