Abstract
This paper establishes the first theoretical guarantees for learning a single neuron with non-monotonic activation functions, such as SiLU and GELU, using gradient descent. While traditional deep learning theory relies on monotonicity to ensure a well-behaved optimization landscape, modern architectures increasingly utilize non-monotonic activations. By analyzing the optimization landscape through the lens of Hermite expansions and empirical process theory, this work proves that as long as an activation function possesses a "dominating" linear or monotonic component, gradient descent converges to the global optimum in polynomial time and with a polynomial number of samples.
Introduction
In theoretical deep learning, the "single neuron" model ($x \to \sigma(w^T x)$) serves as a fundamental building block. Historically, researchers have proven the efficiency of learning this model under the assumption that the activation function $\sigma$ is monotonic (e.g., ReLU or Sigmoid). Monotonicity acts as a safety net, preventing the loss landscape from developing "bad" local minima that could trap optimization algorithms.
However, state-of-the-art models like GPT-4 and BERT have shifted toward non-monotonic activations such as SiLU (Swish) and GELU. These functions exhibit a characteristic "dip" before ascending, theoretically breaking the monotonicity safety net. This paper bridges the gap between practice and theory, providing a rigorous framework to explain why gradient descent remains effective for these modern activation functions.
Analyzing the Optimization Landscape
The core of the paper's contribution lies in its analysis of the optimization landscape—the geometric map of the loss function. The author examines this landscape under several critical settings:
- General Input Distributions: By applying a "spread" condition to input data, the research shows that if the monotonic part of the activation dominates the non-monotonic wiggles, the gradient consistently points toward the global optimum.
- Gaussian Inputs and Hermite Expansion: For Gaussian data, the author utilizes Hermite polynomials to decompose the activation function. This fine-grained analysis reveals how different components (linear, quadratic, etc.) influence the search for optimal weights.
- Empirical Process Theory: To bridge the gap between theoretical infinite-data models and finite-sample reality, the author uses empirical process theory to prove that the "jagged" landscape of a finite dataset remains close enough to the smooth theoretical landscape to preserve convergence.
Key Insights: The Power of the Linear Component
A central finding is that high-probability convergence depends heavily on the "linear component" of the activation function, represented by the first Hermite coefficient. If a function behaves "mostly" like a line, gradient descent can navigate the landscape effectively even if local non-monotonicities exist.
The paper mathematically verifies that popular functions like SiLU/Swish and GELU satisfy this "dominating linear part" condition. Conversely, the author provides a counter-example using a Sine activation; if a function oscillates too much, the linear part vanishes, and gradient descent is likely to fail by getting stuck in a local minimum.
Furthermore, the research highlights the efficacy of zero initialization. For many non-monotonic functions, the gradient at exactly zero points directly toward the ground truth weights $w^*$. This effectively reduces a complex high-dimensional optimization problem into a simpler one-dimensional search.
Key Results and Convergence
The theoretical analysis yields several robust results:
- Polynomial Efficiency: For Gaussian inputs, the paper proves that gradient descent learns ground truth weights using only polynomial samples and time.
- Convergence Rates: With zero initialization, the error decreases exponentially ($e^{-Ct}$). With random initialization (specifically Riemannian GD on a sphere), the paper establishes a high probability of success for Gaussian distributions, provided the linear component is sufficiently strong.
- Landscape Stability: Proposition 5.1 demonstrates that the difference between the "true" expected loss and the "sampled" empirical loss vanishes at a rate of approximately $1/\sqrt{n}$, where $n$ is the number of samples.
Limitations and Future Work
While this work provides a vital foundation, it is limited to the single-neuron, realizable case—where labels are perfectly generated by a neuron of the same type. Real-world networks involve millions of interacting neurons, creating significantly more complex saddle points and plateaus. Future research directions include extending this analysis to two-layer networks and the "agnostic" setting, where labels may be noisy or generated by a different underlying process.
Summary updated