Abstract
This study derives an exact analytical formula for the population gradient—the average gradient over the entire input distribution—for a two-layered ReLU network. Utilizing a student-teacher framework, the analysis reveals that "bad" critical points in the loss landscape typically constitute continuous manifolds rather than isolated points. This geometric property provides a theoretical basis for the "flat minima" observed in deep learning. Additionally, the paper explains how "Spontaneous Symmetry Breaking" (SSB) enables simple gradient descent to overcome non-convexity by allowing neurons to specialize.
Introduction: The Challenge of Non-Convexity
A central mystery in deep learning is the efficacy of gradient descent on highly non-convex loss landscapes. Traditional theoretical models often simplify these landscapes by assuming independent neuron activations or using smooth activation functions like sigmoids. However, the Rectified Linear Unit (ReLU) is non-smooth, and in practical networks, neuron activations are coupled through shared inputs.
To address these complexities, this work adopts a "student-teacher" setup. In this paradigm, a student network attempts to learn the parameters of a fixed teacher network with an identical architecture. By examining the population gradient, we can understand the fundamental "flow" of learning independent of specific batch noise.
Methodology: Deriving the Population Gradient
The analysis focuses on a two-layered ReLU network where the first-layer weights are trainable and the second-layer weights are fixed (e.g., to unity). The primary technical contribution is the derivation of an exact analytical formula for the expected gradient, assuming inputs follow a zero-mean spherical Gaussian distribution $N(0, I)$.
The population gradient is expressed as: $$E[F(e, w)] = \frac{N}{2\pi} [(\pi - \theta)w + \|w\| \sin \theta e]$$ where $\theta$ represents the angle between the current weight vector $w$ and the target control signal $e$. This formula decomposes the gradient into two components: a "mass term" aligned with the current weights and an "asymmetric term" aligned with the target signal.
Theoretical Insights: Manifolds and Symmetry
Critical Points as Manifolds
A significant finding is that critical points (where the gradient is zero) located outside the principal hyperplane spanned by the teacher's weights are not isolated. Instead, they form continuous manifolds. This provides a mathematical explanation for "flat minima," where perturbations to weights do not significantly alter the loss, a property often associated with better generalization.
Spontaneous Symmetry Breaking (SSB)
In multi-node networks, the gradient field is inherently symmetric. If a network is initialized perfectly symmetrically, it may become trapped at a saddle point. However, the paper demonstrates that an infinitesimal perturbation—a "nudge"—triggers Spontaneous Symmetry Breaking. This process causes nodes to "specialize," diverging from their initial symmetric state to explain different components of the teacher signal.
Results and Convergence
The paper establishes concrete bounds for convergence. For a single ReLU node, a weight initialization radius $r \leq \frac{2\pi}{d+1} \|w^*\|$ guarantees convergence to the global optimum with high probability (nearly 50% for random initialization near the origin). This result provides theoretical support for common practices like He or Xavier initialization. Furthermore, simulations of the derived 2D nonlinear dynamics confirm that nodes often take "detours" during training to ensure proper specialization and error reduction.
Conclusion and Limitations
While this work provides a rare closed-form look into ReLU gradient dynamics, it relies on the assumption of spherical Gaussian inputs. Preliminary empirical tests suggest the formula's relevance to other distributions, but formal proofs for non-Gaussian data are still needed. Additionally, extending this analysis to deeper architectures and networks with trainable second layers remains a significant challenge for future research.
Summary updated