Barron Approximation and locally optimal weight densities for shallow neural networks

G. WelperarXiv 2026 · arXiv

TL;DR

This paper introduces a modified "mean field" loss function that directly optimizes the expected sampling error of finite-sized shallow neural networks. By proving this loss is convex and leads to "equidistribution"—where every neuron contributes equally to the approximation—the author shows that practical gradient descent with dropout naturally achieves theoretically optimal Barron-type error bounds.

What Problem Does This Paper Address?

In the world of neural network theory, Mean Field Theory is a powerful tool. It treats a neural network not as a collection of discrete neurons, but as a continuous distribution (a "cloud") of weights. While this works beautifully for infinitely wide networks, translating these results back to the finite networks we actually train is tricky.

The core challenge is the sampling error. If you have an ideal continuous distribution of weights, how do you pick $m$ neurons so that the approximation error is as small as possible? Standard theory gives a rate of $m^{-1/2}$, but this depends heavily on the "Barron norm" of the function, which can explode if your weight distribution is poorly chosen. The paper seeks a way to mathematically guarantee that our optimization methods find the "best" possible weight distribution for finite networks.

Methods

The author proposes a modified mean field loss. Instead of just minimizing the distance between the continuous "cloud" and the target function, the new loss minimizes the expected error of a network sampled from that cloud.

Key technical components include:

  • Modified Loss Function: A new objective that places the expectation outside the norm, providing direct control over finite-sized samples.
  • Two-Timescale Limit: A framework where outer weights (the linear layer) are assumed to be optimal for any given set of inner weights (the hidden features).
  • Wasserstein Gradient Flow (WGF): The author uses WGF to describe how the weight distribution evolves during training.
  • Particle Approximation: To make the theory practical, the continuous flow is discretized into "particles" (actual neurons), which turns the abstract math into standard gradient descent.

Datasets

The paper uses a synthetic benchmark for its numerical experiments:

  • Target Function: $f(x, y) = x^2$ on the domain $[-1, 1]^2$.
  • Significance: Even though the input is 2D, the function is intrinsically 1D. This allows the author to test if the optimization algorithm can "discover" the low-dimensional structure of the data.

Key Insights and Ideas

  • Equidistribution is the "Aha!" Moment: The paper proves that local minimizers of the new loss satisfy an "equidistribution property." This means that in an optimal network, the "importance" of every neuron (the product of its outer weight and the magnitude of its feature) is roughly constant. No single neuron does all the heavy lifting, and no neuron is useless.
  • Convexity in the Large: Despite the non-linear nature of neural networks, the author proves that this specific "expected error" loss is convex over the space of probability distributions. This implies that local minima are actually global minima in the mean field limit.
  • Dropout is Mathematically Natural: One of the most striking findings is that when you discretize this new loss to train a real network, it naturally results in dropout-type regularization. Dropout isn't just a clever trick to prevent overfitting; it is the direct algorithmic consequence of optimizing for expected sampling error.

Key Results

  • Optimal Error Rates: The author proves that networks sampled from the minimizers of this loss achieve an approximation error of $O(m^{-1/2})$, matching the best theoretical bounds known for Barron spaces.
  • Dimensionality Discovery: In experiments, the network successfully aligned its weights with the $x$-axis (the intrinsic dimension of $f(x,y)=x^2$), but only when dropout was used.
  • Equidistribution Plateaus: Numerical results confirmed the theory by showing that neurons' contributions eventually settle into distinct "plateaus"—either they are active and contributing equally, or they are deactivated (zeroed out).

Limitations and Future Work

  • Shallow Networks Only: The current theory is strictly for shallow (one hidden layer) networks. Extending this to deep architectures remains a significant challenge.
  • Auxiliary Width Requirements: Some of the "practical" variants of the outer weights require an auxiliary width ($M$) that is much larger than the actual network width ($m$) for the math to stay clean.
  • Discrete Regularity: While the continuous theory is robust, the author notes that proving the same level of "equidistribution" for strictly discrete sets of points (rather than continuous densities) requires extra regularity proofs that are left for future research.

Summary updated

Abstract

Mean field theories provide optimization results for shallow neural networks by analyzing the weight distribution in infinite width limits. Corresponding results for finite sized networks are obtained by particle approximations, for which sharp quantitative bounds are still an open problem. In this paper, we consider a modified mean field loss, which allows a more fine grained control over finite sized networks. We prove convexity and equidistribution properties, which directly lead to Barron type approximation results for networks sampled from local loss minimizers. We demonstrate that particle approximations of the new loss function naturally lead to gradient descent methods with dropout type regularization.
BibTeX
@misc{welper2026barron,
  author = {G. Welper},
  title = {{Barron Approximation and locally optimal weight densities for shallow neural networks}},
  year = {2026},
  eprint = {2611.19798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19798},
}
Figure