Electron-Proton Dynamics in Deep Learning

Qiuyi Zhang, Rina Panigrahy, Sushant SachdevaarXiv 2017 · OpenReview

Abstract

We study whether a depth two neural network can learn another depth two network using gradient descent. Assuming a linear output node, we show that the question of whether gradient descent converges to the target function is equivalent to the following question in electrodynamics: Given $k$ fixed protons in $\mathbb{R}^d,$ and $k$ electrons, each moving due to the attractive force from the protons and repulsive force from the remaining electrons, whether at equilibrium all the electrons will be matched up with the protons, up to a permutation. Under the standard electrical force, this follows from the classic Earnshaw's theorem. In our setting, the force is determined by the activation function and the input distribution. Building on this equivalence, we prove the existence of an activation function such that gradient descent learns at least one of the hidden nodes in the target network. Iterating, we show that gradient descent can be used to learn the entire network one node at a time.

Introduction

Despite the massive success of deep learning, our theoretical understanding of why it works remains incomplete. The loss landscapes of neural networks are non-convex, which theoretically suggests they should be riddled with local minima that trap gradient descent. This paper addresses the "proper learning" problem: can a neural network learn another network of the same architecture? While this occurs in practice, proving it mathematically is difficult. We seek to identify the conditions under which gradient descent is guaranteed to converge to the global optimum for depth-two networks.

The Electron-Proton Analogy

The core contribution of this work is a conceptual framework called Electron-Proton Dynamics. We demonstrate that the squared loss function of a depth-two network with a linear output is mathematically equivalent to a potential energy function in electrodynamics. In this analogy:

  • The hidden units of the target network act as fixed protons.
  • The hidden units of the learning network act as moving electrons.
  • Gradient descent is the force that moves these "electrons" toward the "protons."

Drawing on Earnshaw’s Theorem—a 19th-century physics principle stating that a collection of point charges cannot be maintained in a stable stationary equilibrium—we look for activation functions that create "harmonic" loss landscapes. If the Laplacian of the loss is non-zero, stable local minima cannot exist, facilitating convergence to the target nodes.

Theoretical Guarantees and Activation Engineering

We prove the existence of activation functions specifically designed to have a "harmonic" potential. For such functions, any stable critical point must be one where at least one learned node matches a target node. While the primary proof uses a complex, non-explicit construction, we also provide results for more standard activations like sign and polynomial functions.

Table summarizing activation functions, their associated potentials, and convergence results.

To leverage these properties, we introduce a "Node-wise Descent Algorithm" that learns the target network one hidden unit at a time. By "capturing" one proton at a time, the algorithm eventually reconstructs the entire target function.

Empirical Validation and the "Depth Wall"

The study validates these claims using synthetic data with inputs sampled from a standard Gaussian distribution. Experiments tested networks with widths ranging from 5 to 125 hidden nodes.

Training error over iterations for networks with widths ranging from 5 to 125.

In simulations, depth-two networks consistently converged to a squared $\ell_2$ error of less than 0.002, regardless of the number of hidden nodes. However, the experiments also revealed a sharp decline in learnability beyond depth two. For a depth-3 network, the error jumped significantly, and for depth-17, it reached 0.577. This suggests that the "harmonic" properties of the landscape that facilitate optimization at depth two break down as networks get deeper.

Limitations and Future Work

While the electron-proton analogy provides powerful geometric intuition, several challenges remain. The theoretical analysis assumes an infinite sample limit and relies on Gaussian input distributions, which rarely occur in real-world datasets. Furthermore, extending this analogy to deep architectures (depth > 2) remains a major open challenge, as the pairwise interactions between nodes become significantly more complex. Future work will focus on relaxing these assumptions and scaling the framework to deeper models.

Summary updated

Abstract

We study whether a depth two neural network can learn another depth two network using gradient descent. Assuming a linear output node, we show that the question of whether gradient descent converges to the target function is equivalent to the following question in electrodynamics: Given $k$ fixed protons in $\mathbb{R}^d,$ and $k$ electrons, each moving due to the attractive force from the protons and repulsive force from the remaining electrons, whether at equilibrium all the electrons will be matched up with the protons, up to a permutation. Under the standard electrical force, this follows from the classic Earnshaw's theorem. In our setting, the force is determined by the activation function and the input distribution. Building on this equivalence, we prove the existence of an activation function such that gradient descent learns at least one of the hidden nodes in the target network. Iterating, we show that gradient descent can be used to learn the entire network one node at a time.
BibTeX
@misc{zhang2017electron,
  author = {Qiuyi Zhang and Rina Panigrahy and Sushant Sachdeva},
  title = {{Electron-Proton Dynamics in Deep Learning}},
  year = {2017},
  eprint = {1702.00458},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.00458},
}
Figure