Abstract
We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor defined by the network. For $L$-layer linear tensor networks that are orthogonally decomposable, we show that gradient flow on separable classification finds a stationary point of the $\ell_{2/L}$ max-margin problem in a "transformed" input space defined by the network. For underdetermined regression, we prove that gradient flow finds a global minimum which minimizes a norm-like function that interpolates between weighted $\ell_1$ and $\ell_2$ norms in the transformed input space. Our theorems subsume existing results in the literature while removing standard convergence assumptions. We also provide experiments that corroborate our analysis.
Introduction
In modern deep learning, models are often "overparameterized," meaning they have far more parameters than training examples. Mathematically, this implies an infinite number of ways to achieve zero training error. Despite this, training with Gradient Descent (GD) typically results in models that generalize well to unseen data. This phenomenon is known as implicit bias: the optimization algorithm itself acts as a regularizer, picking "simple" solutions even without explicit penalties like weight decay.
Previously, researchers studied this bias on a case-by-case basis for different architectures. This paper addresses the lack of a unified framework, providing a single mathematical language—Linear Tensor Networks—to describe how architecture and depth shape this bias.
Methods: Linear Tensor Networks
The authors propose representing network operations as multilinear products of tensors. This allows different architectures to be treated as special cases of a single general form. The framework relies on a Data Tensor (M) that encodes input data and connectivity (e.g., the sliding window of a convolution).
The analysis focuses on "gradient flow" (the continuous-time limit of GD) and a class of networks where the data tensor is orthogonally decomposable. This approach allows the authors to characterize the convergence direction of network parameters as singular vectors of the tensor defined by the network architecture.
Key Insights and Results
The research demonstrates that network architecture effectively defines a "transformed" input space. The implicit bias of GD is essentially a max-margin or min-norm problem being solved within this hidden space.
The Power of Depth and Initialization
For classification, the bias is toward a stationary point of an $\ell_{2/L}$ max-margin problem. As depth $L$ increases, the "quasi-norm" $\ell_{2/L}$ behaves more like an $\ell_1$ norm, explaining why deeper networks often favor sparser solutions. In regression tasks, the final solution is highly sensitive to the starting scale ($\alpha$):
- Small $\alpha$: The network behaves like a sparse $\ell_1$ minimizer.
- Large $\alpha$: The network behaves like a dense $\ell_2$ (Euclidean) minimizer.
Experimental Validation
Using synthetic toy datasets and controlled regression tasks, the authors show that even when different architectures (Convolutional, Diagonal, Fully-connected) start at the same point, their implicit bias pulls them toward completely different global minima, exactly as the tensor theory predicts.
Discussion and Limitations
While this framework successfully subsumes previous results for diagonal and convolutional networks, it is currently restricted to linear networks without nonlinear activations like ReLUs. Additionally, the strongest results require the data tensor to be "orthogonally decomposable," which may not hold for every complex architecture. Future work will focus on extending this unified tensor-based view to nonlinear models and more complex loss functions.
Summary updated