Introduction
Despite the massive success of deep learning, explaining why simple algorithms like gradient descent (GD) work so well on highly complex, non-convex optimization problems remains a significant theoretical challenge. Theoretically, gradient descent should frequently get stuck in local minima or stall at saddle points.
Previous attempts to prove convergence have often relied on the "independent activation" assumption—the idea that a neuron's activation is independent of its input. However, this is physically impossible in real networks. This paper provides a rigorous theoretical analysis of how gradient descent navigates the non-convex landscape of two-layer ReLU networks without relying on unrealistic independence assumptions. By treating the learning process as a dynamical system, the author proves that while symmetric initializations lead to dead-end saddle points, "symmetry-breaking" initializations allow the network to achieve global convergence to the optimal solution.
Methodology: A Dynamical Systems Approach
To tackle this problem, the author analyzes the nonlinear dynamics of ReLU networks under a "teacher-student" framework. In this setup, a student network attempts to mimic a fixed teacher network using realistic Gaussian input distributions.
The author employs Dynamical Systems theory to model the weight updates of the network as a first-order differential equation ($dw/dt = -\nabla E(w)$). Key methodological contributions include:
- ReLU Gradient Formulation: The paper derives a concise rule for gradient updates in ReLU networks by noting that the Jacobian of a ReLU layer is "transparent"—it acts as a diagonal gating matrix that only depends on the direction of the weights, not their magnitude.
- Expected Gradient Analysis: Instead of looking at individual noisy updates, the author calculates the expected gradient over a Gaussian input distribution $N(0, I)$. This effectively "smooths out" the non-differentiable point of the ReLU at zero.
- Lyapunov Stability: To prove convergence, the author uses Lyapunov functions (a standard tool in control theory) to demonstrate that the distance between the student's weights and the teacher's weights strictly decreases over time under certain conditions.
Key Insights: The Power of Asymmetry
The analysis reveals several fascinating dynamics about how neural networks learn:
- The Symmetry Trap: If a student network is initialized with perfectly symmetric weights, the gradient updates for all neurons remain identical. This forces the network into a "saddle point"—a state where it is stuck and cannot capture the complexities of the data.
- Spontaneous Symmetry Breaking: Drawing a parallel to physics, the paper shows that a tiny "nudge" (perturbation) away from symmetry allows the neurons to "specialize." Once symmetry is broken, one neuron becomes the "leader," and the others follow to explain the remaining components of the data.
- Initialization Matters: The paper mathematically verifies why common initialization techniques (like He or Glorot initialization) are effective. It shows that for a single ReLU node, a random initialization near the origin has at least a 50% chance of falling into the "convergence basin" of the global optimum.
- Detours are Necessary: Simulations show that even when a student is perfectly aligned with the teacher, the dynamics often take a "detour" (temporarily increasing error) to allow neurons to specialize before converging to the final solution.
Theoretical Results
The paper provides concrete proofs for different network sizes:
- Global Convergence for $K=1$: For a single-neuron network, the system is proven to be asymptotically stable, converging to the teacher's weights $w^*$ if initialized within a specific region.
- Global Convergence for $K \ge 2$: When the teacher weights are orthonormal, a symmetry-breaking initialization leads to global convergence. This stands as one of the first proofs of global convergence in nonlinear networks that accounts for the true dependency of ReLU activations.
- Closed-Form Dynamics: The author provides an exact 2D nonlinear dynamical equation describing weight evolution, which matches empirical simulations with high precision.
Limitations and Future Work
While groundbreaking, the theoretical framework has current limitations. The proof for multiple neurons ($K \ge 2$) assumes that the teacher's weight vectors are orthonormal (perpendicular and of unit length), which is rarely the case in real-world scenarios. Additionally, the analysis assumes identical architectures for both the student and teacher, leaving "over-parameterized" cases unexplored.
Finally, the closed-form solutions rely on the mathematical properties of Gaussian distributions. While simulations suggest these findings hold for uniform distributions, a formal proof for general distributions is still needed. Future work will aim to extend this "symmetry-breaking" framework to deeper, multi-layer networks and more complex activation functions.
Summary updated