Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

Alexandre Declèves, Etienne Boursier, Nicolas FlammarionarXiv 2026 · arXiv

TL;DR

Fine-tuning is the backbone of modern AI, yet we lack a clear theory for why starting from pretrained weights is so much more efficient than starting from scratch. This paper uses diagonal linear networks to prove that pretraining reshapes the "implicit bias" of gradient descent, allowing models to recover a sparse signal using far fewer samples by focusing only on the coordinates that were missing or incorrectly signed during pretraining.

What Problem Does This Paper Address?

In the "pretrain-then-fine-tune" paradigm, we assume that knowledge from a source task helps us learn a target task faster. While this works brilliantly in practice, theoretical machine learning has historically focused on "training from scratch" with small, uninformative initializations.

The authors tackle a fundamental gap: How exactly does a structured initialization (pretrained weights) reduce the sample complexity of a downstream task? Specifically, they look at sparse recovery—identifying which few features out of thousands are actually relevant—and ask if fine-tuning can "inherit" the correct features while efficiently learning the ones it missed, all without getting distracted by noise or spurious features.

Methods

The authors study this through the lens of Diagonal Linear Networks (DLNs). These are simple two-layer networks ($y = (u \odot v)^\top x$) that, despite being linear in their end-to-end mapping, possess non-convex optimization dynamics that naturally favor sparse solutions.

Key components of their approach include:

  • Mirror Flow Analysis: They characterize the training trajectory as a "mirror flow" toward a specific implicit regularizer.
  • Saddle-to-Saddle Dynamics: They analyze the limit where the network's initialization is very small (vanishing imbalance). In this regime, training behaves like a discrete sequence of "jumps," where the model activates one feature at a time.
  • Weighted Lasso Benchmark: They compare the performance of gradient-based fine-tuning against a "Weighted Lasso" estimator, which is a standard statistical tool that explicitly uses prior knowledge to penalize different features differently.
  • Null-Gradient Stopping Rule: They propose a data-dependent way to stop training early, preventing the model from overfitting to noise after it has recovered the true signal.

Datasets

The paper is primarily theoretical. To validate their proofs, they use synthetic datasets for sparse linear regression:

  • Design Matrices: Generated as i.i.d. Gaussian matrices ($X \in \mathbb{R}^{n \times d}$).
  • Target Vectors: Sparse vectors $\beta^*$ where only $s$ out of $d$ coordinates are non-zero.
  • Pretrained Initialization: A vector $\beta^0$ that shares some (but not necessarily all) of the support and signs of the target $\beta^*$.

Key Insights and Ideas

  • Implicit Bias is Sign-Aware: Unlike training from zero, which treats positive and negative weights equally, fine-tuning creates an asymmetric penalty. It is "cheaper" for the model to keep a feature with its pretrained sign than to flip it or activate a brand-new feature.
  • The Geometry of Recovery: The authors visualize the "distance" a feature must travel in the dual space to become active.
  • Correctly signed features start at the finish line (distance 0).
  • Missing features start in the middle (distance 1).
  • Wrongly signed features start at the opposite end and must travel twice as far (distance 2).
Figure 1: Schematic dual trajectories under non-zero initialization. Missing and wrongly signed true coor- dinates travel dual distances 1 and 2, respectively, before entering with the correct sign.
  • Sample Complexity Gains: The "cost" of learning (in terms of required data) is shifted. Instead of paying a logarithmic penalty for every relevant feature, you only pay it for the features that were missing or wrong in the pretrained model.
  • Persistence of False Positives: A surprising finding is that the implicit bias of fine-tuning does not naturally "clean up" false positives inherited from pretraining. If the pretrained model thought a feature was important and it wasn't, fine-tuning tends to keep it unless the noise level is extremely low.

Key Results

  • Noiseless Exact Recovery: In a setting without noise, the authors prove that fine-tuning converges to the true signal with a sample size of roughly $n \gtrsim s + f_0 + m \log(d)$, where $m$ is the number of missing/wrongly signed features and $f_0$ is the number of inherited false positives. This is a significant improvement over the standard $s \log(d)$ requirement for training from scratch.
  • Noisy Recovery via Early Stopping: In the presence of noise, the model will eventually overfit. However, the authors show that their "null-gradient" stopping rule can stop the trajectory at a point where all true features are recovered, but no new false positives have been added.
  • Comparison to Lasso: Fine-tuning achieves a sample complexity comparable to the Weighted Lasso, proving that gradient descent implicitly performs the same kind of sophisticated feature selection that statisticians do explicitly.

Limitations and Future Work

  • Gaussian Assumption: The current proofs rely on the design matrix being i.i.d. Gaussian. Real-world data is often highly correlated, which changes the optimization landscape.
  • Vanishing Imbalance Limit: The analysis assumes the "imbalance" between the two layers of the DLN is near zero. While this simplifies the math, practical deep learning uses finite initializations.
  • Feature Learning: This model uses a fixed basis (linear regression). A major next step would be extending this to non-linear "feature learning" regimes, where the model can actually transform the input space during fine-tuning.

Summary updated

Abstract

Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuning leverages information from pretraining remains poorly understood theoretically. We study fine-tuning from pretrained weights through the lens of sparse linear regression and two-layer diagonal linear networks. In our setting, pretraining provides information through the support (and signs) of the initialization predictor, which may contain coordinates relevant to the downstream task. We show how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support. In particular, for a clean initialization with correctly inherited signs, we show that the required sample size is comparable to that of a weighted Lasso estimator that explicitly exploits the pretrained support through a suitably chosen regularizer. Our results thus show how information encoded in pretrained weights can be implicitly exploited by gradient-based fine-tuning, reducing the amount of data needed to recover a downstream task.
BibTeX
@misc{decleves2026statistical,
  author = {Alexandre Declèves and Etienne Boursier and Nicolas Flammarion},
  title = {{Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks}},
  year = {2026},
  eprint = {2609.34756},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.34756},
}
Figure