When does Gaussian equivalence fail and how to fix it: Non-universal behavior of random features with quadratic scaling

Garrett G. Wen, Hong Hu, Yue M. Lu, Zhou Fan, Theodor MisiakiewiczarXiv 2025 · DOI · arXiv

Introduction

In high-dimensional statistics, Gaussian Equivalence Theory (GET) serves as a powerful heuristic. It posits that complex machine learning models with many features can be analyzed by replacing those features with Gaussian random variables that match the first two moments. While GET has successfully explained phenomena like "double descent" in linear scaling regimes ($n \asymp p \asymp d$), it encounters significant hurdles in more complex settings.

The Breakdown in Quadratic Scaling

A recent study by Wen et al. identifies a fundamental failure of GET in the quadratic scaling regime, where the number of samples $n$ and features $p$ grow proportionally to the square of the data dimension $d$ ($n \asymp p \asymp d^2$). In this regime, if the target function possesses a "low-dimensional structure"—such as a single-index model—GET's predictions can be fundamentally incorrect.

The failure is particularly acute when using non-quadratic training or test losses (e.g., hinge or 0-1 loss). In these cases, structured "spikes" in the data do not average out into a Gaussian background, leading to a breakdown in universality.

Comparison of test error predictions between standard Gaussian Equivalence (GE) and the proposed Conditional Gaussian Equivalence (CGE) across different loss regimes.

The Conditional Gaussian Equivalent (CGE) Model

To address this, the authors introduce the Conditional Gaussian Equivalent (CGE) framework. Instead of a blanket Gaussian substitution, CGE conditions on the low-dimensional signal subspace. It treats the high-dimensional "background" as Gaussian while preserving the non-Gaussian nature of the structured signal.

This hybrid approach is supported by a rigorous mathematical foundation involving:

  • Wiener Chaos Expansion: Decomposing random features into Hermite polynomials to distinguish noise from signal.
  • Two-Phase Lindeberg Swapping: A technique to systematically replace real features with CGE surrogates.
  • Malliavin-Stein Argument: Establishing a quantitative Central Limit Theorem for these complex expansions.

Key Insights: Structure and Loss

The research reveals that universality is sensitive to both data structure and the choice of loss function.

  1. Structure is the Enemy of Universality: Structured (low-rank) signals create non-Gaussian "spikes" that standard GET cannot replicate.
  2. Loss Function Sensitivity: Interestingly, GET remains accurate even in the quadratic regime if both training and test losses are quadratic ($\ell_2$). The failure is triggered by non-quadratic losses typical in classification tasks.
Distribution of the random feature predictor's output. (a) Breakdown of Gaussianity for structured single-index models. (b) Maintenance of Gaussianity for unstructured random-polynomial models.

Conclusion

The CGE model provides the first sharp asymptotic predictions for random feature models in the quadratic scaling regime. By "appending" a small number of non-Gaussian features to a Gaussian base, it restores theoretical tractability to complex modern models. Future work aims to extend this "Conditional Universality" principle to even higher-order scalings and more diverse architectures.

Summary updated

Abstract

A major effort in modern high-dimensional statistics has been devoted to the analysis of linear predictors trained on nonlinear feature embeddings via empirical risk minimization (ERM). Gaussian equivalence theory (GET) has emerged as a powerful universality principle in this context: it states that the behavior of high-dimensional, complex features can be captured by Gaussian surrogates, which are more amenable to analysis. Despite its remarkable successes, numerical experiments show that this equivalence can fail even for simple embeddings -- such as polynomial maps -- under general scaling regimes. We investigate this breakdown in the setting of random feature (RF) models in the quadratic scaling regime, where both the number of features and the sample size grow quadratically with the data dimension. We show that when the target function depends on a low-dimensional projection of the data, such as generalized linear models, GET yields incorrect predictions. To capture the correct asymptotics, we introduce a Conditional Gaussian Equivalent (CGE) model, which can be viewed as appending a low-dimensional non-Gaussian component to an otherwise high-dimensional Gaussian model. This hybrid model retains the tractability of the Gaussian framework and accurately describes RF models in the quadratic scaling regime. We derive sharp asymptotics for the training and test errors in this setting, which continue to agree with numerical simulations even when GET fails. Our analysis combines general results on CLT for Wiener chaos expansions and a careful two-phase Lindeberg swapping argument. Beyond RF models and quadratic scaling, our work hints at a rich landscape of universality phenomena in high-dimensional ERM.
BibTeX
@misc{wen2025when,
  author = {Garrett G. Wen and Hong Hu and Yue M. Lu and Zhou Fan and Theodor Misiakiewicz},
  title = {{When does Gaussian equivalence fail and how to fix it: Non-universal behavior of random features with quadratic scaling}},
  year = {2025},
  eprint = {2512.03325},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.03325},
}
Figure