Propagation of Chaos in One-hidden-layer Neural Networks beyond Logarithmic Time
Lists
Only you see this
Abstract
We study the approximation gap between the dynamics of a polynomial-width neural network and its infinite-width counterpart, both trained using projected gradient descent in the mean-field scaling regime. We demonstrate how to tightly bound this approximation gap through a differential equation governed by the mean-field dynamics. A key factor influencing the growth of this ODE is the local Hessian of each particle, defined as the derivative of the particle's velocity in the mean-field dynamics with respect to its position. We apply our results to the canonical feature learning problem of estimating a well-specified single-index model; we permit the information exponent to be arbitrarily large, leading to convergence times that grow polynomially in the ambient dimension $d$. We show that, due to a certain ``self-concordance'' property in these problems -- where the local Hessian of a particle is bounded by a constant times the particle's velocity -- polynomially many neurons are sufficient to closely approximate the mean-field dynamics throughout training.
BibTeX
@misc{glasgow2025propagation,
author = {Margalit Glasgow and Denny Wu and Joan Bruna},
title = {{Propagation of Chaos in One-hidden-layer Neural Networks beyond Logarithmic Time}},
year = {2025},
eprint = {2504.13110},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2504.13110},
}Loading related content...