Global convergence of neuron birth-death dynamics

Grant M. Rotskoff, Samy Jelassi, Joan Bruna, Eric Vanden-EijndenarXiv 2019 · arXiv

Introduction

In the optimization of neural networks, gradient descent (GD) serves as the standard workhorse. However, GD is fundamentally a local update rule; it moves parameters incrementally along the loss landscape. In complex models, this can lead to agonizingly slow convergence or cause the network to become trapped in suboptimal regions, particularly if the initialization is poor. This paper explores a novel optimization framework that moves beyond local transport by introducing a birth-death dynamic. By allowing neurons to be removed or cloned based on their performance, the authors demonstrate that networks can achieve non-local jumps in parameter space, providing both a theoretical guarantee of global convergence and significant empirical speedups.

The Problem: Locality in the Mean-Field Limit

To analyze neural network behavior, researchers often look at the "mean-field" limit, where a network is viewed as a continuous distribution of particles (neurons) rather than a finite set of weights. Under standard gradient descent, this distribution evolves according to a deterministic partial differential equation (PDE) known as a Wasserstein gradient flow.

While mathematically elegant, this flow is strictly local. If a neuron is initialized in a region of the parameter space where it contributes little to the function approximation, it must slowly traverse the landscape to reach a useful configuration. If the initial distribution fails to cover critical regions, the network may stall at spurious fixed points. The authors address this by asking whether the fundamental dynamics can be modified to allow parameters to "teleport" to where they are most needed while maintaining mathematical rigor.

The Birth-Death Framework

The authors propose augmenting the standard gradient flow with a birth-death process. This mechanism actively manages the population of neurons through four key steps:

  1. Evaluation: The "potential" of each neuron is calculated, representing its specific contribution to the overall loss.
  2. Death: Neurons located in ineffective regions (high potential) are removed with a probability proportional to their lack of utility.
  3. Birth (Cloning): Neurons in highly effective regions (low potential) are duplicated.
  4. Conservation: To ensure the model remains manageable, the total number of neurons is strictly conserved, balancing the birth and death rates.

For practical application in finite networks, this is implemented by interleaving standard Stochastic Gradient Descent (SGD) steps with a resampling step. The authors also introduce a "reinjection" variant, where new neurons can be sampled from a prior distribution rather than just cloning existing ones, further increasing the diversity of the parameter search.

Theoretical Insights

The paper provides several rigorous proofs that elevate the birth-death dynamic from a heuristic to a robust optimization theory:

  • Global Optimality: Unlike standard GD PDEs, which can suffer from local minima, the authors prove that the only fixed point of the birth-death PDE is the global minimum of the loss function.
  • Self-Quenching Noise: A common concern with stochastic birth-death processes is the introduction of excessive noise. The authors demonstrate that the variance of this process is proportional to the distance from the optimum. As the network approaches the global minimum, the fluctuations naturally vanish—a property termed "self-quenching."
  • Information Geometry: The birth-death step is shown to be mathematically equivalent to a Kullback-Leibler (KL) proximal step. This bridges the gap between optimal transport (moving mass via GD) and information geometry (updating distributions).
  • Convergence Rates: The authors derive asymptotic rates, showing exponential convergence for non-interacting parameters and an $O(t^{-1})$ rate for interacting parameters typical of neural networks.

Empirical Results

The framework was validated on synthetic tasks designed to test the limits of optimization:

  • Mixture of Gaussians: In 1D and 2D tasks, the birth-death dynamics significantly outperformed standard gradient descent in reaching the target distribution.
  • Student-Teacher ReLU Networks: In a 50-dimensional task where a 50-neuron student network approximated a 10-neuron teacher, the birth-death approach demonstrated much faster convergence.
  • Robustness to Initialization: In scenarios with deliberately poor initializations (where neurons were clustered far from the target), standard GD failed entirely. The birth-death dynamics, particularly with the reinjection variant, successfully spawned neurons in the correct regions to solve the task.

Conclusion and Future Directions

By integrating birth-death dynamics into the mean-field limit of neural networks, the authors provide a powerful tool to bypass the limitations of local gradient flows. While this work focuses on optimization and single-hidden-layer networks, it opens several avenues for future research. Key questions remain regarding how these dynamics influence the generalization gap on unseen data and how the framework can be extended to deep, multi-layer architectures. Additionally, optimizing the "prior" for neuron reinjection could further enhance the ability of networks to escape complex, high-dimensional local minima.

Summary updated

Abstract

Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models. In this regime, gradient descent obeys a deterministic partial differential equation (PDE) that converges to a globally optimal solution for networks with a single hidden layer under appropriate assumptions. In this work, we propose a non-local mass transport dynamics that leads to a modified PDE with the same minimizer. We implement this non-local dynamics as a stochastic neuronal birth-death process and we prove that it accelerates the rate of convergence in the mean-field limit. We subsequently realize this PDE with two classes of numerical schemes that converge to the mean-field equation, each of which can easily be implemented for neural networks with finite numbers of parameters. We illustrate our algorithms with two models to provide intuition for the mechanism through which convergence is accelerated.
BibTeX
@misc{rotskoff2019global,
  author = {Grant M. Rotskoff and Samy Jelassi and Joan Bruna and Eric Vanden-Eijnden},
  title = {{Global convergence of neuron birth-death dynamics}},
  year = {2019},
  eprint = {1902.01843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.01843},
}
Figure