Deep Learning via Neural Energy Descent

Wenrui Hao, Chunmei Wang, Xingjian Xu, Haizhao YangarXiv 2023 · OpenReview

Abstract

This paper proposes the Neural Energy Descent (NED) via neural network evolution equations for a wide class of deep learning problems. We show that deep learning can be reformulated as the evolution of network parameters in an evolution equation and the steady state solution of the partial differential equation (PDE) provides a solution to deep learning. This equation corresponds to a gradient descent flow of a variational problem and hence the proposed time-dependent PDE solves an energy minimization problem to obtain a global minimizer of deep learning. This gives a novel interpretation and solution to deep learning optimization. The computational complexity of the proposed energy descent method can be enhanced by randomly sampling the spatial domain of the PDE leading to an efficient NED. Numerical examples are provided to demonstrate the numerical advantage of NED over stochastic gradient descent (SGD).

Introduction

Training deep neural networks is notoriously difficult because the "loss landscape"—the mathematical map of errors—is incredibly rugged and non-convex. Standard tools like Stochastic Gradient Descent (SGD) are often likened to a hiker trying to find the lowest valley in a mountain range during a fog; they frequently get stuck in local minima or wander aimlessly.

Furthermore, in scientific computing, we often need to solve high-dimensional partial differential equations (PDEs). Traditional numerical methods (like finite elements) fail here because of the "curse of dimensionality," where computational cost explodes as dimensions increase. While neural networks are adept at handling high dimensions, training them to solve PDEs accurately remains a significant challenge for standard optimizers. This paper introduces Neural Energy Descent (NED), a novel framework that treats deep learning as a physical evolution process toward a global minimum.

Methodology: The Evolutionary Approach

Instead of viewing training as a series of discrete, heuristic steps, NED models it as a continuous evolution equation. The training process is rewritten as a time-dependent PDE where the network parameters $\theta(t)$ evolve over "time."

Figure 1: Comparison of continuous and discrete gradient flows in function and parameter spaces. The diagram illustrates how the discrete flow over parameters approximates the continuous flow toward the optimal solution $u^*$.

As illustrated in Figure 1, the goal is to reach a "steady state" where the parameters stop changing—this state corresponds to the optimal solution. This evolution follows a "gradient descent flow" of a specific energy functional. In physics terms, it is akin to a system naturally moving toward its lowest energy state.

To implement this, the authors utilize sophisticated numerical tools:

  • Higher-Order Solvers: While standard SGD is essentially a crude 1st-order approximation (Forward Euler), NED can employ the 2nd-order Runge-Kutta (RK2) method. This significantly improves precision and helps bypass the performance plateaus common in standard deep learning.
  • Moore-Penrose Inverse: To determine the exact "flow" of parameters, the algorithm solves a least-squares problem at each step, often utilizing the Moore-Penrose pseudoinverse to handle under-determined systems.

Key Results and Insights

The paper evaluates NED on various scientific machine learning tasks, including synthetic multi-dimensional functions and complex nonlinear boundary value problems (BVPs) in up to 30 dimensions.

  • Superior Accuracy: In almost every test, NED (especially the RK2 version) achieved significantly lower relative $L^2$ errors than SGD. For example, in learning a 1D sine wave, NED-RK2 reached error levels orders of magnitude lower than SGD.
  • Stability in High Dimensions: For a 30-dimensional quadratic function, NED maintained a steady downward trajectory in error, while SGD hit a performance ceiling much earlier.
  • PDE Solving: When solving 5D nonlinear BVPs, NED-RK2 converged to a much more accurate solution than SGD-based physics-informed methods, which struggled to capture the fine details of the steady-state solution.

Conclusion and Future Work

The "aha moment" of this research is realizing that a trained model is simply the steady-state solution of a specific physical process. This allows researchers to borrow centuries of wisdom from PDE theory to improve deep learning optimization.

While NED shows great promise, challenges remain regarding computational cost. Solving the least-squares problem at each step is more expensive per iteration than a simple SGD step. Future work will explore scaling NED to larger models and investigating how well these models generalize to unseen data in classic machine learning tasks. Nonetheless, NED provides a stronger theoretical foundation and a more robust path to global minimizers than current heuristic approaches.

Reference: Hao, W., Wang, C., Xu, X., & Yang, H. (2023). Deep Learning via Neural Energy Descent. arXiv:2302.10424v1 [math.NA].

Summary updated

Abstract

This paper proposes the Nerual Energy Descent (NED) via neural network evolution equations for a wide class of deep learning problems. We show that deep learning can be reformulated as the evolution of network parameters in an evolution equation and the steady state solution of the partial differential equation (PDE) provides a solution to deep learning. This equation corresponds to a gradient descent flow of a variational problem and hence the proposed time-dependent PDE solves an energy minimization problem to obtain a global minimizer of deep learning. This gives a novel interpretation and solution to deep learning optimization. The computational complexity of the proposed energy descent method can be enhanced by randomly sampling the spatial domain of the PDE leading to an efficient NED. Numerical examples are provided to demonstrate the numerical advantage of NED over stochastic gradient descent (SGD).
BibTeX
@misc{hao2023deep,
  author = {Wenrui Hao and Chunmei Wang and Xingjian Xu and Haizhao Yang},
  title = {{Deep Learning via Neural Energy Descent}},
  year = {2023},
  eprint = {2302.10424},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.10424},
}
Figure