Adaptive activation functions accelerate convergence in deep and physics-informed neural networks

Ameya D. Jagtap, Kenji Kawaguchi, George Em KarniadakisJournal of Computational Physics 2020 · OpenReview

TL;DR

This paper introduces an "adaptive activation function" for neural networks, which includes a tunable hyper-parameter. This simple modification significantly speeds up training and improves accuracy for both standard deep learning tasks and, crucially, for Physics-Informed Neural Networks (PINNs) solving complex differential equations.

What Problem Does This Paper Address?

Training deep neural networks, especially for complex tasks like solving partial differential equations (PDEs) with Physics-Informed Neural Networks (PINNs), can be slow and prone to getting stuck in suboptimal solutions. Traditional activation functions (like ReLU, tanh, sigmoid) are fixed, meaning their behavior is predetermined. This inflexibility can hinder the network's ability to efficiently learn the intricate patterns present in data, particularly when dealing with high-gradient solutions or when trying to discover underlying physical laws from data. The paper aims to improve both the speed and accuracy of neural network training by making the activation function itself more adaptable.

Methods

The core innovation is the introduction of a scalable hyper-parameter, denoted by 'a', into the activation function. This parameter, along with a scaling factor 'n', allows the activation function's slope to be dynamically adjusted during training. Instead of being fixed, the activation function's shape can change as the network learns, guided by the optimization process. This adaptive nature allows the network to better navigate the loss landscape, avoiding local minima and accelerating convergence. The authors apply this adaptive activation function to standard deep neural networks and, more importantly, to PINNs. In PINNs, the loss function is augmented with terms that enforce the governing physical equations, and the adaptive activation function helps the network satisfy these constraints more efficiently.

Datasets

The paper uses several standard datasets for evaluating general deep learning performance, including:

  • CIFAR-10
  • CIFAR-100
  • SVHN
  • MNIST
  • KMNIST
  • Fashion-MNIST
  • Semeion

For the PINN applications, the "datasets" are generated from the exact solutions of the PDEs being solved, or from high-resolution numerical solutions. These include:

  • Nonlinear Klein-Gordon equation
  • Nonlinear Burgers equation
  • Helmholtz equation
  • Two-dimensional sine-Gordon equation (for inverse problems)

Key Insights and Ideas

  • Adaptive Activation Functions are a Simple Yet Powerful Improvement: The paper demonstrates that a minor modification to activation functions can lead to significant gains in training speed and solution accuracy.
  • Faster Convergence for PINNs: The adaptive activation function dramatically accelerates the convergence of PINNs, which is crucial for solving complex differential equations.
  • Improved Frequency Capture: By analyzing solutions in the frequency domain, the authors show that adaptive activation functions can capture high-frequency components of solutions much faster than fixed ones. This is particularly important for problems with sharp gradients or discontinuities.
  • Robustness to Noise: The adaptive approach shows improved accuracy and parameter identification even when the training data contains noise, as demonstrated in the inverse problem settings.
  • Dynamic Loss Landscape Navigation: The adaptive hyper-parameter allows the network to dynamically adjust its learning strategy, effectively changing the "topology" of the loss function to find better solutions more efficiently.

Key Results

  • Accelerated Convergence: The adaptive activation function consistently leads to faster convergence across all tested problems, both for standard deep learning benchmarks and for PINN applications. For instance, in the Klein-Gordon equation, adaptive activation captured all frequencies in 22,000 iterations compared to fixed activation.
  • Increased Accuracy: The adaptive approach generally yields more accurate solutions, with lower relative L2 errors reported for various PDEs. For the Sine-Gordon inverse problem, the adaptive activation function achieved significantly lower relative L2 errors in parameter identification, especially in the presence of noise.
  • Effective for High-Gradient Problems: The Burgers equation, known for its steep gradients, showed substantial improvements in solution accuracy and frequency capture with the adaptive activation.
  • Efficient Parameter Identification: In inverse problems, the adaptive PINN approach accurately identified unknown parameters of the governing equations, even with noisy data.

Limitations and Future Work

  • Hyper-parameter Tuning: While the adaptive activation function itself is tuned, the scaling factor 'n' and the initial value of 'a' might still require some experimental tuning. The paper notes that using a very large scaling factor 'n' can lead to oscillations and sensitivity, suggesting that regularization might be needed.
  • Theoretical Guarantees: The paper mentions theoretical proof that gradient descent algorithms are not attracted to suboptimal critical points or local minima with the proposed method, which is a significant theoretical contribution. However, further exploration of the theoretical underpinnings of this adaptive approach could be beneficial.
  • Broader Applicability: While the paper focuses on specific PDEs, future work could explore the application of adaptive activation functions to a wider range of complex scientific and engineering problems. The authors suggest that this approach is promising for both PINNs and standard neural networks, implying broad applicability.

Summary updated

Abstract

Highlights • We employed adaptive activation functions in deep and physics-informed neural networks. • The proposed method is very simple and it is shown to accelerate convergence in neural networks. • In particular, we approximate various nonlinear functions using deep neural networks. • Physics-informed neural networks are employed to solve both forward and inverse problems of PDEs. • We also solved standard deep learning benchmarks problems and theoretically proved convergence results. Abstract We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima. Furthermore, the proposed adaptive activation functions are shown to accelerate the minimization process of the loss values in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion datasets with and without data augmentation.
BibTeX
@article{jagtap2020adaptive,
  author = {Ameya D. Jagtap and Kenji Kawaguchi and George Em Karniadakis},
  title = {{Adaptive activation functions accelerate convergence in deep and physics-informed neural networks}},
  journal = {Journal of Computational Physics},
  year = {2020},
  url = {https://openreview.net/forum?id=JFCJuDOWWs},
}
Figure