Evolutionary weight initialization runs a population-based search, such as a genetic algorithm or the evolution strategy CMA-ES, to pick a network's starting weights before ordinary gradient training takes over. It is usually justified by an "initialization lottery": some random draws start in bad regions of the loss surface. Infinite-width theory seems to contradict that story. In the mean-field view, a wide layer is a cloud of neurons described by a probability distribution, and training moves that distribution along a smooth path that, for two-layer networks, reaches a global optimum. If the lottery disappears as width grows, what is evolution buying?
We propose that the answer is exact and testable. When evolution searches over individual weights, it exploits the random fluctuations of a finite network around its infinite-width limit. Those fluctuations shrink like $1/\sqrt{m}$ in the width $m$, so the benefit should vanish for wide networks. When evolution searches over the distribution that neurons are drawn from, it changes the limit itself, so its benefit should survive at any width. Because the quality of a distribution can be judged from quantities that stabilize quickly with width, the search can run on narrow proxy networks and be transferred to wide ones at no extra cost. The project builds this "measure-genome" search, tests the predicted width scaling against weight-level evolution, and maps where it helps and where it provably cannot.
Core Claim
A width-$m$ two-layer network under mean-field scaling outputs the average of $m$ neuron contributions, each drawn independently from an initial distribution $\mu_0$. Propagation-of-chaos results bound how far such a network sits from its infinite-width limit. Improved Particle Approximation Error for Mean Field Neural Networks bounds the loss gap by the prediction variance, which is $O(1/N)$ in the number of neurons $N$.
This bound has a consequence for selection. Picking the best of $P$ independent random draws is the core of any evolutionary initializer operating on weights. Its expected fitness gain behaves like the maximum of $P$ roughly Gaussian fluctuations:
$$\Delta_{\text{weights}}(m, P) \approx \sigma_m \sqrt{2 \ln P}, \qquad \sigma_m = c/\sqrt{m},$$
Here $\sigma_m$ is the standard deviation of early training loss across random draws at width $m$, and $c$ is a task-dependent constant. The $\sqrt{2\ln P}$ factor is the standard extreme-value growth for Gaussian maxima. Mutation and crossover at a fixed evaluation budget should not change the $m^{-1/2}$ rate, because they too act on fluctuations rather than on the limit.
Now let the genome be a low-dimensional parameter $\phi$ of the initial distribution $\mu_{0,\phi}$ itself. Its effect on the limiting training trajectory does not depend on $m$. The separation-of-scales result of On the Provable Separation of Scales in Maximal Update Parameterization says that macro-quantities such as expected loss converge across width at $O(1/n)$, while individual weights drift at $\Theta(1/\sqrt{n})$. In that language, weight-level evolution optimizes a micro-variable and measure-level evolution optimizes a macro-variable. Only the latter should transfer across width.
The claim therefore makes three predictions. Weight-level evolutionary gains decay as $m^{-1/2}$. Measure-level gains are flat in $m$. Genome rankings computed at small width predict rankings at large width with error shrinking roughly as $1/m$.
Relation to Prior Work
Three papers motivate the gap. The scaling path paper, On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks, shows that the initialization scale $\alpha$ moves a two-layer network continuously from feature learning to lazy kernel behavior. It keeps the base distribution fixed (uniform on a sphere or Gaussian). We ask whether shape descriptors of that distribution matter beyond $\alpha$.
M2N2, from Competition and Attraction Improve Model Fusion, evolves networks from scratch on MNIST using split points inside flat weight arrays, plus fitness sharing to keep the population diverse. It is our prototype of weight-level evolution; it has no width analysis. The ELM study, Forecasting short-term electric load using extreme learning machine with improved tree seed algorithm based on Lévy flight, shows large gains from evolving the frozen hidden weights of small extreme learning machines. Our account predicts exactly this regime, small $m$ with no training of the hidden layer, is where the lottery is largest.
Two papers constrain the design. Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input proves that for targets with a "reflective" symmetry, gradient flow from a symmetric initial distribution stays stuck for any constant time. This implies that any rotation-invariant genome cannot fix such targets in the wide limit. We state that limitation as a prediction rather than hide it.
On the mean field limit of consensus based methods proves a mean-field limit for consensus-based optimization, a swarm method related to particle swarm optimization. That result lets us use the swarm as the outer search over genomes with a well-posed large-population limit.
One paper supplies a component. EvoGrad: Evolutionary-Weighted Gradient and Hessian Learning for Black-Box Optimization combines CMA-ES with learned local gradients. It is a candidate outer optimizer when the genome grows beyond a handful of dimensions.
Research Program
Everything is defined under mean-field (maximal-update) scaling for two-layer networks, where the theory is sharpest, followed by a depth extension on CIFAR-10.
The first component is the genome. We use a rotation-invariant genome so that search cannot quietly learn task directions. Its entries are the overall scale $\alpha$, a radial profile for input-weight norms (a two-parameter generalized Gaussian, with shape controlling tail heaviness), the ratio of output-weight to input-weight scale, a correlation between a neuron's input norm and its output magnitude, and the fraction of neurons initialized with near-zero output weight. That last entry is the "mass" knob, echoing the Hellinger part of the unbalanced transport metric in the scaling path paper.
We also define an anisotropic genome that adds a low-rank covariance. It is used only in the symmetry experiments, and its search samples are charged to its data budget.
The second component is the fitness and search. Fitness is validation loss after a short, fixed training horizon at a proxy width $m_{\text{proxy}}$, averaged over a few seeds. The average is used because it is a macro-quantity, which is what the separation-of-scales argument licenses. The outer search is CMA-ES or consensus-based optimization over the roughly six-dimensional genome.
The third component is analysis. We will attempt to prove the weight-level decay rate for best-of-$P$ selection under the variance bound of Nitanda's paper. We will also prove that rotation-invariant initial distributions preserve reflective symmetry along the whole mean-field trajectory, which extends the Chen and Ge barrier to every rotation-invariant genome.
Baselines
The naive reference is standard maximal-update initialization: Gaussian weights with the default scale, trained by plain gradient descent. It receives the same total compute that the evolutionary methods spend on search plus training, used as extra training steps and a learning-rate sweep. Beating it means a lower final test loss at matched total compute.
The strongest baseline on the core question is the $\alpha$-only genome. This is one-dimensional search over initialization scale, which is precisely the knob On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks identifies as controlling the regime. It gets the same outer optimizer, proxy width and evaluation budget as the full genome. The full measure-genome must beat it by more than seed-to-seed spread at the target width. Otherwise the richer genome is unnecessary and the contribution reduces to "tune $\alpha$ on a small model."
The strongest existing evolutionary competitor is weight-level evolution, in two forms. One is M2N2 from Competition and Attraction Improve Model Fusion. The other is flat-vector CMA-ES. Both run at each width with an identical number of fitness evaluations. On MNIST from scratch the file reports that M2N2 beat CMA-ES but gives no number (no reported result found on Lacuna for M2N2 MNIST accuracy), so we reproduce both.
On the frozen-hidden-layer regime, the reference is the ELM result of Forecasting short-term electric load using extreme learning machine with improved tree seed algorithm based on Lévy flight. On EUNITE it reports 96.60% accuracy and MSE 0.00212 for its evolved initialization, against 92.20% for a standard ELM. We rerun a standard ELM, weight-level evolution and the measure-genome, sweeping the number of hidden units upward from the paper's setting.
A noise baseline, mean-field Langevin training with tuned temperature, isolates whether any gain simply comes from exploration during training.
Why not just…?
Why not just tune the initialization scale $\alpha$? This is the most serious threat. The scaling path result says $\alpha$ alone sets the rich-versus-lazy regime, and it may carry most of the effect. The $\alpha$-only ablation above settles the question directly. If the full genome's advantage over $\alpha$-only is within noise at target width on the single-index and CIFAR-10 tasks, we will report that result. The proposal then narrows to its width-scaling claim about weight-level evolution, which stands independently.
Why not just evolve the weights of a small network directly, as M2N2 or the ELM work does? Those methods work at small width. Our prediction is that their gain shrinks as $m^{-1/2}$ and does not transfer, because individual weights are micro-variables in the separation-of-scales sense of On the Provable Separation of Scales in Maximal Update Parameterization. The width sweep tests this directly. A flat gain curve for weight-level evolution would falsify the core claim.
Why not just add Langevin noise during training? Noise acts during training rather than on the starting point. Improved Particle Approximation Error for Mean Field Neural Networks notes that its time to converge can depend exponentially on the log-Sobolev constant at low temperature. That concern is theoretical, and a tuned temperature might still match a good initial distribution at moderate cost. The Langevin baseline is compute-matched, so this is an empirical question, not something we assume away.
Why not just initialize neurons along data principal directions? That approach is anisotropic and uses data, so it belongs with the anisotropic genome and is compared against it. On reflective-symmetry targets, both are charged the samples they consume.
Proposed Experiments
The first decisive check is the width sweep. On the quartic single-index task on the sphere from Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time, on MNIST, and on a two-layer CIFAR-10 network up to width 10,000 as in the separation-of-scales paper, we measure the gain over the naive baseline for widths from 64 to 10,000. We compare weight-level evolution against the measure-genome. For the single-index task, the file reports the sample-complexity separation of $O(d^{3.1})$ for mean-field gradient flow versus $\Omega(d^4)$ for kernels. We record the empirical samples needed to reach fixed error under each initialization, to test whether any genome shifts that scaling. The prediction is a log-log slope near $-1/2$ for weight-level gains and near zero for measure-level gains.
The second check is transfer. We evolve genomes at widths 128 and 512, deploy them unchanged at 8,192, and report two quantities. One is the rank correlation of genome fitness across widths. The other is the fraction of the gain found by searching directly at the target width that transfer retains. We expect cross-width rank disagreement to fall roughly as $1/m$.
The third check is the symmetry barrier. On subspace-sparse Gaussian polynomials with and without the reflective property, from Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input, rotation-invariant genomes should fail on reflective targets at large width and succeed on non-reflective ones. Only the anisotropic genome, charged for its samples, should break the barrier. On the 2D interpolation task of the scaling path paper, we compare decision boundaries from evolved genomes against the fixed-distribution path to see which shape descriptors move the network off that path. The EUNITE ELM sweep checks the small-width, frozen-layer end of the same curve, where weight-level evolution should win.
References
- Improved Particle Approximation Error for Mean Field Neural Networks
- On the Provable Separation of Scales in Maximal Update Parameterization
- On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks
- Competition and Attraction Improve Model Fusion
- Forecasting short-term electric load using extreme learning machine with improved tree seed algorithm based on Lévy flight
- Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input
- On the mean field limit of consensus based methods
- EvoGrad: Evolutionary-Weighted Gradient and Hessian Learning for Black-Box Optimization
- Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time