Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

Hien Dang, Pratik Patil, Alessandro Rinaldo

Self-distillation (SD) is a fascinating phenomenon in machine learning where a model acts as its own teacher. By retraining a "student" model on a mixture of ground-truth labels and the teacher's own predictions—using the exact same architecture and training data—practitioners often observe a "free lunch" in the form of improved generalization. However, despite its empirical success, the formal guarantees explaining why and when self-distillation works have remained limited.

This paper establishes a rigorous theoretical foundation for self-distillation in ridge regression. By allowing the mixing weight between ground-truth labels and teacher predictions to be unconstrained, the authors prove that a student model can strictly outperform its teacher at almost any regularization level. Furthermore, they introduce a highly efficient "one-shot" tuning method that achieves near-optimal performance without the need for expensive grid searches.

The Core Problem: Understanding the "Free Lunch"

Most existing theories on self-distillation are constrained to specific settings or assume that the mixing weight ($\xi$) must be a convex combination, strictly bounded between 0 and 1. In practice, finding the optimal mixing weight typically requires an exhaustive and computationally expensive grid search.

This paper addresses these gaps by analyzing self-distillation in the context of ridge regression (a standard linear model with $L_2$ regularization) and removing the traditional bounds on the mixing weight.

Methodology: Unconstrained Mixing and Asymptotics

The authors introduce several novel analytical elements to formalize the benefits of self-distillation:

  1. Unconstrained Mixing: Unlike standard distillation, the target is not restricted to a convex combination. By allowing $\xi \in \mathbb{R}$, the model can engage in "anti-learning" (correcting for high noise) or "pro-learning" (correcting for over-regularization).
  2. Structural Identities: The research derives exact formulas for the optimal mixing weight ($\xi^\star$) and the resulting risk, establishing a direct mathematical link between the optimal weight and the derivative of the teacher's risk.
  3. Proportional Asymptotics: The model is analyzed in a high-dimensional regime where the number of features ($p$) and samples ($n$) both approach infinity at a fixed ratio, providing precise risk asymptotics for general anisotropic covariance and deterministic signals.

Key Insights and Theoretical Guarantees

The theoretical analysis yields several striking insights into the mechanics of self-distillation:

  • Pointwise Strict Improvement: The most significant finding (Theorem 2.2) is that if a teacher model is not already perfectly tuned—meaning its risk derivative is non-zero ($R'(\lambda) \neq 0$)—an optimally mixed student is mathematically guaranteed to strictly improve upon the teacher. Self-distillation acts as a localized corrective mechanism rather than just a global heuristic.
  • The Sign Rule: The optimal mixing weight $\xi^\star$ naturally takes a sign opposite to the teacher's risk derivative. If the teacher is under-regularized, $\xi$ is positive. Crucially, if the teacher is over-regularized, $\xi$ becomes negative. This negative weight effectively "undoes" the teacher's excessive shrinkage.
  • A "Cheap" Regularization Path: Self-distillation allows the model to implicitly traverse its regularization path. It can elevate a poorly tuned teacher to perform almost as well as a perfectly tuned one, achieving near-Bayes optimal risk in certain scenarios.
  • Same-X vs. Fresh-X: Counter-intuitively, distilling on the same training features often yields better results than using new, unlabeled "fresh" features. The coupling between the teacher and student on identical data proves to be highly beneficial.

Practical Application: One-Shot Tuning

From a practical standpoint, the authors propose a consistent "one-shot" tuning method based on Generalized Cross-Validation (GCV). This method estimates the optimal mixing weight $\xi^\star$ directly from the training data. It completely bypasses the need for grid search, sample splitting, or refitting the model multiple times, offering a massive computational speedup while accurately tracking true test risk.

Empirical Validation

The theoretical claims and the one-shot tuning method are validated across a diverse set of datasets:

  • Real-World Regression: UCI BlogFeedback and UCI Communities and Crime.
  • Out-of-Distribution (OOD) Robustness: UCI Air Quality, demonstrating that the strict improvements hold even for OOD data.
  • Deep Learning Features: CIFAR-10 and CIFAR-100 image classification tasks, applying ridge regression to features extracted from pretrained ResNet-18 and ResNet-34 models.
  • Synthetic Data: Used to verify the precise asymptotic formulas under various signal-to-noise ratios and covariance structures.

Limitations and Future Directions

While the paper provides robust guarantees, the core proofs rely on the linearity of ridge regression and are specific to squared prediction risk. Although empirical success is shown on neural network features, the theory does not yet fully encompass non-linear deep learning architectures or other loss functions like cross-entropy. Future work will need to explore how this "regularization path geometry" persists in more complex models and whether similar one-shot tuning guarantees can be developed for broader distillation techniques.

Summary updated

Abstract

Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth labels and the teacher’s own predictions using the same architecture and training data. Although SD has been empirically shown to often improve generalization, its formal guarantees remain limited. We study SD for ridge regression with an unconstrained mixing weight $\xi \in \mathbb{R}$. Conditional on the training data and without any distributional assumptions, we prove that for any squared prediction risk $R$ (including out-of-distribution), the optimally mixed student strictly improves upon the ridge teacher at every regularization level $\lambda$ where the teacher risk is not stationary ($R'(\lambda) \neq 0$). We also characterize the optimal mixing weight $\xi^\star$ in terms of the risk derivative $R'$, showing that it can be negative, which is the case in over-regularized regimes. To quantify SD risk improvements, we derive exact risk asymptotics in the proportional asymptotics regime for general anisotropic covariance and deterministic signals. From a practical standpoint, we propose a consistent one-shot tuning method to estimate $\xi^\star$ without grid search, sample splitting, or refitting. Experiments on real-world datasets and pretrained neural network features support our theory and the one-shot tuning method.
BibTeX
@inproceedings{dang2026optimal,
  author = {Hien Dang and Pratik Patil and Alessandro Rinaldo},
  title = {{Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=MdHcU4C4Rm},
}
Figure