Incorporating priors in learning: a random matrix study under a teacher-student framework

Malik Tiomoko, Ekkehard SchnoorarXiv 2025 · DOI · arXiv

Introduction

Modern machine learning rarely begins with a blank slate. Practitioners frequently leverage "priors"—initial weight estimates derived from pre-trained models, temporal patterns, or domain-specific constraints. Despite their ubiquity, the theoretical understanding of how these informative priors interact with high-dimensional data has remained incomplete.

In the "proportional regime," where the number of features ($d$) is comparable to the number of samples ($n$), classical statistical intuition often falters. This regime is characterized by phenomena such as "double descent," where test error spikes near the interpolation threshold ($d \approx n$). While standard Ridge regression (assuming a zero-centered prior) has been extensively studied, this work provides the first exact mathematical characterization of Maximum A Posteriori (MAP) regression with an informative prior centered at a non-zero guess ($\Theta_0$).

Theoretical Framework and Methods

The authors utilize a Teacher-Student framework to model the learning process. A "teacher" generates data using a hidden weight matrix $\Theta_\star$, while a "student" attempts to recover these weights using a MAP estimator that penalizes the distance from a prior guess $\Theta_0$.

The technical backbone of this study is Random Matrix Theory (RMT). By employing "deterministic equivalents," the authors derive closed-form expressions for training and test risks in the limit where $d, n \to \infty$ with a fixed ratio $c = d/n$. The analysis assumes data features follow "q-exponentially concentrated" distributions, a broad class encompassing Gaussians and features generated by Generative Adversarial Networks (GANs).

Theoretical risk curves (solid lines) match empirical simulations (shaded regions), validating the RMT-based formulas across different regularization strengths $\lambda$.

Key Insights: The Bias-Variance-Prior Tradeoff

The study extends the traditional bias-variance tradeoff by introducing the prior as a third critical pillar. A high-quality prior reduces the bias introduced by regularization, shifting the optimal balance for the regularization parameter $\lambda$.

A significant finding is the mitigation of the "double descent" phenomenon. The interpolation peak at $d \approx n$ is primarily driven by variance. An informative prior can substantially suppress this peak, although the underlying mathematical structure of the double descent remains. The authors identify a "variance inflator" term, $1-\alpha$, which approaches zero as $d \approx n$ under low regularization, causing risk to explode. A good prior effectively manages this instability.

Test and train risk as a function of the aspect ratio $c=d/n$. An informative "good" prior significantly suppresses the interpolation peak compared to the "no prior" case.

Results and Optimal Regularization

The paper provides exact asymptotic formulas for test risk ($R_{test}$) and training risk ($R_{train}$) as functions of noise variance ($\sigma^2$), prior mismatch ($S = \|\Theta_\star - \Theta_0\|^2$), and the data aspect ratio ($c$).

For the underparameterized regime ($c < 1$), the authors derive an elegant formula for the optimal regularization parameter: $$\lambda^\star = \frac{\sigma^2}{S} \cdot \frac{1}{1-c}$$

Crucially, the authors demonstrate a data-driven method to estimate the unknown parameters $\sigma^2$ and $S$ directly from training error at extreme values of $\lambda$. This allows practitioners to determine the optimal regularization strength without a separate validation set, purely from training data.

Conclusion and Future Work

While this work offers a rigorous foundation for understanding priors in high-dimensional linear regression, several avenues for future research remain. The current closed-form solutions for optimal $\lambda$ and parameter estimation strictly apply to identity covariance matrices ($\Sigma = I$); general covariance structures present more complex signal decompositions. Additionally, extending this framework to non-linear models and non-Gaussian priors would bridge the gap toward modern deep learning architectures.

Summary updated

Abstract

Regularized linear regression is central to machine learning, yet its high-dimensional behavior with informative priors remains poorly understood. We provide the first exact asymptotic characterization of training and test risks for maximum a posteriori (MAP) regression with Gaussian priors centered at a domain-informed initialization. Our framework unifies ridge regression, least squares, and prior-informed estimators, and -- using random matrix theory -- yields closed-form risk formulas that expose the bias-variance-prior tradeoff, explain double descent, and quantify prior mismatch. We also identify a closed-form minimizer of test risk, enabling a simple estimator of the optimal regularization parameter. Simulations confirm the theory with high accuracy. By connecting Bayesian priors, classical regularization, and modern asymptotics, our results provide both conceptual clarity and practical guidance for learning with structured prior knowledge.
BibTeX
@misc{tiomoko2025incorporating,
  author = {Malik Tiomoko and Ekkehard Schnoor},
  title = {{Incorporating priors in learning: a random matrix study under a teacher-student framework}},
  year = {2025},
  eprint = {2509.22124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22124},
}
Figure