Int-Deep: A Deep Learning Initialized Iterative Method for Nonlinear Problems

Jianguo Huang, Haoqin Wang, Haizhao YangJournal of Computational Physics 2019 · DOI · OpenReview

TL;DR

This paper introduces "Int-Deep," a novel hybrid approach that combines deep learning with traditional iterative methods to efficiently solve nonlinear partial differential equations (PDEs). By using deep learning to provide a high-quality initial guess, Int-Deep significantly accelerates the convergence of standard numerical solvers, achieving high accuracy with reduced computational cost.

What Problem Does This Paper Address?

Solving nonlinear partial differential equations (PDEs) is a fundamental challenge in science and engineering. Traditional numerical methods like finite difference, spectral, and finite element methods can be computationally expensive, especially for nonlinear problems, as they often require solving large systems of nonlinear equations. A critical bottleneck is finding a good initial guess for iterative solvers (like Newton's method) to ensure convergence to the correct solution. While deep learning has shown promise in solving PDEs, its efficiency in low-dimensional settings compared to traditional methods is still unclear, often requiring many iterations or yielding limited accuracy. This paper aims to bridge this gap by leveraging the strengths of both deep learning and traditional iterative methods.

Methods

The proposed "Int-Deep" framework operates in two phases:

  • Phase I: Deep Learning Initialization: This phase uses deep neural networks (DNNs) to approximate the solution space of a given nonlinear PDE. The authors formulate the problem as an expectation minimization problem, which can be efficiently solved using stochastic gradient descent (SGD). This approach is designed to be computationally inexpensive, requiring only O(1) computational cost relative to the final accuracy, and O(100) epochs for training. The output is an approximate solution, $u_{DL}$.
  • Phase II: Traditional Iterative Refinement: The approximate solution $u_{DL}$ from Phase I is then used as an initial guess for traditional iterative methods (e.g., Newton's method for PDEs, power method for eigenvalue problems). This hybrid approach aims to achieve high accuracy with a significantly reduced number of iterations, typically O(log(1/ε)), where ε is the desired precision. The framework is designed to handle various types of nonlinear problems, including variational equations, variational inequalities, and eigenvalue problems.

Datasets

Not discussed in the paper. The paper focuses on numerical methods and their theoretical analysis, using synthetic problems for demonstration rather than real-world datasets.

Key Insights and Ideas

  • Synergy of Deep Learning and Traditional Methods: The core idea is that deep learning excels at providing a good "starting point" (initial guess) for traditional iterative solvers, which then refine this guess to achieve high accuracy efficiently. This avoids the need for extensive deep learning training or the challenges of finding good initial guesses for traditional methods alone.
  • Expectation Minimization for Variational Problems: The authors propose a unified variational framework for solving nonlinear PDEs, including those involving variational inequalities and eigenvalue problems, by reformulating them as expectation minimization problems solvable by SGD. This broadens the applicability of deep learning for PDE solving.
  • Computational Efficiency: The deep learning phase is designed to be computationally cheap relative to the final accuracy, and the subsequent iterative phase converges rapidly, leading to an overall efficient solver.
  • Theoretical Guarantees: The paper provides theoretical analysis to justify the convergence of the Int-Deep framework, offering a principled design methodology rather than relying solely on empirical results.

Key Results

The paper demonstrates through numerical experiments that the Int-Deep framework:

  • Outperforms existing methods: It achieves better accuracy and/or faster convergence compared to purely deep learning-based methods or traditional iterative methods alone.
  • Achieves high accuracy: The method can converge to solutions with the precision of finite element methods.
  • Robustness to DNN size: The performance of Int-Deep is shown to be largely independent of the size (width and depth) of the DNN used in Phase I, as long as it's trained sufficiently.
  • Efficient convergence: Traditional iterative methods, when initialized by Int-Deep, converge in a small number of iterations (e.g., O(log(1/ε))). For instance, in Example 5.4, the convergence order of Newton's method was verified to be close to 2, achieving high accuracy with a few iterations.

Limitations and Future Work

  • Low-Dimensional Focus: The current framework is primarily designed for low-dimensional nonlinear PDEs. Extending it to high-dimensional problems remains a challenge.
  • Problem-Dependent Hyperparameters: While the DNN size might not be critical, the hyperparameters for both the deep learning phase (e.g., learning rate, batch size) and the traditional iterative phase (e.g., initial guess quality) can be problem-dependent.
  • Future Work: The authors suggest future work on developing efficient traditional iterative algorithms for variational inequalities, leveraging the good initialization provided by deep learning. They also mention that the analysis of convergence for Algorithm 3 (hybrid Newton's method for nonlinear eigenvalue problems) is left for future work.

Summary updated

Abstract

This paper focuses on proposing a deep learning initialized iterative method (Int-Deep) for low-dimensional nonlinear partial differential equations (PDEs). The corresponding framework consists of two phases. In the first phase, an expectation minimization problem formulated from a given nonlinear PDE is approximately resolved with mesh-free deep neural networks to parametrize the solution space. In the second phase, a solution ansatz of the finite element method to solve the given PDE is obtained from the approximate solution in the first phase, and the ansatz can serve as a good initial guess such that Newton's method for solving the nonlinear PDE is able to converge to the ground truth solution with high-accuracy quickly. Systematic theoretical analysis is provided to justify the Int-Deep framework for several classes of problems. Numerical results show that the Int-Deep outperforms existing purely deep learning-based methods or traditional iterative methods (e.g., Newton's method and the Picard iteration method).
BibTeX
@article{huang2019int,
  author = {Jianguo Huang and Haoqin Wang and Haizhao Yang},
  title = {{Int-Deep: A Deep Learning Initialized Iterative Method for Nonlinear Problems}},
  journal = {Journal of Computational Physics},
  year = {2019},
  doi = {10.1016/j.jcp.2020.109675},
}
Figure