Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves

Giovanni Ballarin, Lukas Gonon, Lyudmila GrigoryevaarXiv 2026 · arXiv

Random Weight Neural Networks (RWNNs)—often called Random Feature Models—are a favorite in the machine learning community because they are incredibly fast to train. By keeping hidden-layer weights random and fixed, we turn a complex neural network problem into a simple linear regression. But there is a catch: as we add more neurons to improve accuracy, the features become highly redundant, leading to "ill-conditioned" math where standard statistical tools start to break down.

In a new paper titled "Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves," authors Giovanni Ballarin, Lyudmila Grigoryeva, and Lukas Gonon provide a rigorous framework for doing reliable statistical inference with these models, even when the underlying matrices are nearly singular.

TL;DR

The paper proves that ill-conditioning is an inevitable structural property of random weight neural networks as they grow. It develops a new theory using ridge regularization and "effective dimension" to provide consistent estimation and valid confidence intervals for these models, even when the feature covariance matrix is singular.

What Problem Does This Paper Address?

In the world of nonparametric statistics, we often use "sieves"—sequences of models that get more complex as we get more data. RWNNs are a type of random sieve. Theoretically, as you add more hidden units ($K$), your approximation error should drop.

However, the authors point out a fundamental gap: standard sieve theory assumes that the features remain "well-conditioned" (meaning they don't become too redundant). In RWNNs, because weights are sampled from a fixed distribution, adding more neurons inevitably leads to overlapping features. This causes the smallest eigenvalue of the population covariance matrix to vanish, making standard OLS estimators explode and rendering traditional confidence intervals useless.

Methods

The authors tackle this by explicitly embracing the ill-conditioning rather than trying to avoid it. Their approach involves several key components:

  • Spectral Analysis: They prove that for common activation functions (like ReLU or tanh) on compact spaces, the eigenvalues of the feature covariance matrix must decay at a polynomial rate.
  • Ridge-Regularized Estimation: Instead of standard least squares, they use ridge regression. They show that the statistical performance is governed not by the number of neurons $K$, but by the ridge effective dimension ($N_{K,\lambda}$), which grows much more slowly.
  • Directional Inference: This is the "aha" moment of the paper. They demonstrate that you don't need the entire covariance matrix to be well-behaved to perform inference. As long as the specific "direction" of the functional you are interested in (e.g., a point prediction or an average effect) is well-conditioned relative to the ridge penalty, you can still get a valid Gaussian limit for your estimates.
  • Advanced Sampling: To improve accuracy, they move beyond simple i.i.d. sampling of weights, utilizing "control-neighbors" and stratified sampling to achieve faster approximation rates than the standard Monte Carlo rate of $1/\sqrt{K}$.
Summary of the paper's main theoretical results across approximation, estimation, and inference.

Datasets

As a theoretical work, the paper relies on synthetic datasets to validate its claims:

  • 1D Regression Benchmarks: They use specific non-linear functions (from Hall and Horowitz, 2013) to test how well their confidence intervals cover the true values.
  • Simulated Environments: They vary the sample size ($n$) from 100 to 3,000 and the number of neurons ($K$) from 15 to 40, testing different activation functions (ReLU, tanh) and weight distributions (Uniform, Gaussian).

Key Insights and Ideas

  • Ill-conditioning is a Feature, Not a Bug: The paper proves that as you increase the resolution of a random feature model, you must encounter redundancy. It’s a direct consequence of the geometry of the parameter space.
  • The Power of Ridge Normalization: By measuring quantities relative to the regularized (ridge) matrix rather than the raw covariance matrix, the authors find a stable way to define "signal" and "noise."
  • Rayleigh Quotient as a Diagnostic: They introduce a ridge-normalized Rayleigh quotient to characterize "directional nondegeneracy." This allows researchers to check if a specific inference task is mathematically "safe" even if the overall model is overparameterized.

Key Results

The paper provides a comprehensive "menu" of convergence rates and inferential limits, summarized in the table below.

Detailed breakdown of spectral decay and estimation conclusions.

Key quantitative findings include:

  • Faster Approximation: Using control-neighbors integration, they achieve $L^2$ approximation rates of $O(K^{-1/2 - \beta/(d+1)})$, which is significantly better than the standard $O(K^{-1/2})$.
  • Valid Inference: They prove that their studentized statistics converge to a standard normal distribution $N(0,1)$. In simulations, their "sandwich-type" variance estimator maintained coverage near the nominal 95% level even as the covariance matrix became singular.
  • Effective Dimension Control: They show that even with 500+ neurons, the "effective" dimension might only be around 50-100, explaining why these models generalize well despite their nominal size.

Limitations and Future Work

While the paper provides a solid foundation, the authors note a few areas for further exploration:

  • Hyperparameter Tuning: The choice of the ridge parameter $\lambda$ is critical. While the theory allows $\lambda$ to vanish, picking the "optimal" $\lambda$ for a specific dataset remains a practical challenge.
  • Bias Correction: In regimes where the ridge penalty is high, the resulting "regularization bias" can shift confidence intervals. Developing automated bias-correction methods is a logical next step.
  • Uniform Bands: The current inference theory focuses on pointwise or functional-specific results. Extending this to "uniform confidence bands" (intervals that hold across the entire function simultaneously) is a stated goal for future research.

Summary updated

Abstract

Random weight neural networks (RWNNs) are shallow networks whose hidden-layer parameters are randomly generated and then kept fixed, while only the output coefficients are estimated. Conditional on the sampled parameters, RWNN regression is a linear sieve problem with random approximation space. We show that such random sieves cannot remain uniformly well-conditioned as their dimension grows. For parametrized feature families satisfying polynomial covering and Lipschitz conditions, the eigenvalues of the population covariance matrix are proven to have polynomial upper bounds determined by the covering geometry of the parameter space, leading to a vanishing smallest eigenvalue. We develop approximation, estimation, and inference theory that explicitly treats the ill-conditioning. Under a $β$-Hölder source condition, we obtain both faster-than-Monte-Carlo $L^2$ approximation rates and new $L^\infty$ approximation bounds based on stratified sampling. We then derive improved consistency guarantees for ridge-regularized least squares driven by ridge effective dimension, spectral, and leverage-based stability conditions. For inference, we establish asymptotic linearity and Gaussian limits for linear functional directions while allowing the smallest eigenvalue of the population covariance matrix to vanish. Nondegeneracy is directional: we characterize it with a ridge-normalized Rayleigh quotient and extend it to smooth functionals via Riesz representation. Consistency of ridge-adjusted variance estimators is proved, yielding feasible studentized inference. Numerical experiments illustrate the deterioration of random-feature conditioning and the finite-sample behavior of the resulting estimation and inference procedures.
BibTeX
@misc{ballarin2026statistical,
  author = {Giovanni Ballarin and Lukas Gonon and Lyudmila Grigoryeva},
  title = {{Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves}},
  year = {2026},
  eprint = {2611.26410},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26410},
}
Figure