Learning a Single Neuron with Bias Using Gradient Descent

Gal Vardi, Gilad Yehudai, Ohad Shamir

Introduction

In the theoretical study of deep learning, the single ReLU neuron is often considered the "hydrogen atom"—the most fundamental unit of analysis. While the optimization of a single neuron without bias ($x \mapsto \sigma(\langle \mathbf{w}, \mathbf{x} \rangle)$) is well-documented, the inclusion of a bias term ($b$) introduces significant complexity. This paper investigates the optimization landscape of a single neuron with bias, $x \mapsto \sigma(\langle \mathbf{w}, \mathbf{x} \rangle + b)$, revealing that this seemingly minor addition fundamentally warps the loss surface and introduces new challenges for gradient-based optimization.

The Challenge of the Bias Term

Adding a scalar bias might appear trivial, but it destroys the spherical symmetry often exploited in convergence proofs. In the bias-less case, the input distribution is typically assumed to be "spread out" (e.g., Gaussian). However, a bias term effectively adds a constant coordinate to the input vector, ensuring the distribution is no longer centered or symmetric in the augmented space.

The authors identify several critical hurdles introduced by bias:

  1. The "Always Off" Trap: If the bias is initialized too negatively, the neuron may output zero for all inputs in the training set. In these "flat" regions, the gradient is zero, leaving the neuron "dead" and unable to learn.
  2. Manifolds of Bad Critical Points: Unlike the bias-less setting, where critical points are usually limited to the global minimum and the origin, the presence of bias creates a continuous manifold of sub-optimal critical points where gradient descent can stall.
  3. Initialization Sensitivity: Standard random initialization can lead to failure with a probability approaching 1/2. This occurs because the optimization often starts in a region where the target neuron's behavior is invisible to the gradient.

Methodology and Theoretical Framework

The study focuses on the realizable setting, where a ground-truth target neuron exists, and the objective is to minimize the squared loss. The researchers employ a combination of geometric and algebraic techniques:

  • Critical Point Characterization: A mathematical mapping of all points where the gradient vanishes to identify non-global minima.
  • Gradient Flow Analysis: Modeling the continuous trajectories of weights to understand the dynamics of training.
  • Algebraic Handling of Distributions: A novel technique to manage non-spherically symmetric distributions, allowing for broader applicability than previous models.

Key Findings and Convergence Results

The paper provides both cautionary results and optimistic guarantees.

Failure and Success Probabilities

The authors prove that for certain target neurons with negative bias, standard gradient descent fails to converge with probability at least $1/2 - o_d(1)$. However, they also demonstrate that if the neuron is initialized such that its starting loss is even slightly better than the "trivial" loss (the loss of a neuron that always outputs zero), convergence to the global optimum is almost guaranteed.

Linear Convergence

Under mild assumptions—specifically using a "smart" initialization where the bias starts at zero—the authors prove linear convergence. This indicates that the error decreases exponentially over time, providing a strong theoretical basis for the efficiency of gradient descent in this context.

Improving Existing Theory

Interestingly, the techniques developed to handle bias also improved the state-of-the-art results for the bias-less case. The researchers increased the guaranteed success probability from approximately $1/2$ to $1 - e^{-\Omega(d)}$, offering a much higher degree of theoretical certainty for high-dimensional problems.

Limitations and Future Directions

While this work advances our understanding of single-neuron dynamics, it is constrained to the realizable setting. It does not account for "agnostic" scenarios involving label noise or cases where the target function is not a single ReLU neuron.

Future research directions include:

  • Extending analysis to Stochastic Gradient Descent (SGD).
  • Scaling the theory to two-layer networks or convolutional architectures with bias.
  • Investigating the "agnostic" setting to better reflect real-world data noise.

Conclusion

By demonstrating how a single scalar term can fundamentally alter the optimization landscape, this research serves as both a cautionary tale and a roadmap. It highlights the necessity of accounting for bias in deep learning theory and provides the mathematical tools needed to guarantee convergence in increasingly complex models.

Summary updated

Abstract

We theoretically study the fundamental problem of learning a single neuron with a bias term ($\mathbf{x}\mapsto \sigma(\langle\mathbf{w},\mathbf{x}\rangle + b)$) in the realizable setting with the ReLU activation, using gradient descent. Perhaps surprisingly, we show that this is a significantly different and more challenging problem than the bias-less case (which was the focus of previous works on single neurons), both in terms of the optimization geometry as well as the ability of gradient methods to succeed in some scenarios. We provide a detailed study of this problem, characterizing the critical points of the objective, demonstrating failure cases, and providing positive convergence guarantees under different sets of assumptions. To prove our results, we develop some tools which may be of independent interest, and improve previous results on learning single neurons.
BibTeX
@inproceedings{vardi2021learning,
  author = {Gal Vardi and Gilad Yehudai and Ohad Shamir},
  title = {{Learning a Single Neuron with Bias Using Gradient Descent}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=m0l7vTv70BK},
}
Figure