Inf-Sup neural networks for high-dimensional elliptic PDE problems

Xiaokai Huo, Hailiang Liu

Solving partial differential equations (PDEs) in high dimensions has long been a "final boss" for numerical analysts. Traditional methods like Finite Element Analysis (FEM) crumble under the "curse of dimensionality" because they rely on meshes that grow exponentially with the number of dimensions. While Physics-Informed Neural Networks (PINNs) have offered a mesh-free alternative, they often struggle with stability and low-regularity solutions.

A new paper by Xiaokai Huo and Hailiang Liu introduces Inf-SupNet, a model-based unsupervised learning approach that reframes elliptic PDEs as a minimax game between two neural networks.

TL;DR

Inf-SupNet is a novel mesh-free neural solver that transforms elliptic PDE problems into a constrained optimization task solved via an adversarial "inf-sup" (minimax) formulation. By using one network to represent the solution and another as a Lagrangian multiplier, the method achieves high accuracy and stability in high-dimensional, nonlinear, and irregular domain problems with provable error bounds.

What Problem Does This Paper Address?

The primary challenge is the curse of dimensionality. If you want to solve a PDE in 10 dimensions using a grid, even a coarse grid with 10 points per dimension requires 10 billion points. Neural networks are naturally "mesh-free" and scalable, making them ideal candidates for high-dimensional physics.

However, existing neural solvers have trade-offs:

  1. PINNs: Often minimize the squared residual of the PDE. This can effectively turn a second-order PDE into a fourth-order optimization problem, which is difficult to solve if the solution isn't perfectly smooth.
  2. Weak Adversarial Networks (WANs): These use a minimax approach but can suffer from numerical instability.

The authors sought a method that maintains the stability of classical "inf-sup" (or Ladyzhenskaya–Babuška–Brezzi) conditions used in traditional finite element theory but adapts them for the deep learning era.

Methods

Inf-SupNet reformulates the PDE $Au = f$ with boundary conditions $Bu = g$ as a constrained optimization problem. Instead of just minimizing a loss, it seeks to minimize the boundary mismatch while treating the PDE itself as a hard constraint.

To solve this, they introduce a Lagrangian multiplier ($v$), turning the problem into a minimax (inf-sup) objective: $$\min_{u} \max_{v} \mathcal{L}(u, v) = \frac{1}{2} \|Bu - g\|^2_{\partial \Omega} + (Au - f, v)_\Omega$$

The Architecture: * The $U$-network ($u_\theta$): Represents the trial solution. * The $V$-network ($v_\tau$): Acts as an adversarial "critic" or Lagrangian multiplier. It tries to find regions where the $U$-network fails to satisfy the PDE.

Training: The networks are trained using Stochastic Gradient Descent-Ascent (SGDA). Points are sampled randomly from the domain and boundary (Monte-Carlo sampling), meaning no mesh is ever required.

Datasets

The authors evaluated Inf-SupNet on several synthetic but challenging benchmarks designed to test specific PDE properties:

  • High-Dimensional Poisson Equations: Tested up to $d=5$ dimensions.
  • Irregular Domains: An L-shaped domain to test handling of non-convex geometries.
  • Mixed Boundary Conditions: Combining Dirichlet and Neumann conditions.
  • Semilinear and Nonlinear PDEs: Testing cases where the operator $A$ depends on the solution $u$ itself (e.g., $q(u) = (1+u)^2$).

Key Insights and Ideas

  • The Duality Gap as a Metric: Unlike standard PINNs where the loss value alone can be misleading, Inf-SupNet uses the "duality gap" (the difference between the max and min steps). This provides a more rigorous quantification of the training error.
  • Error Decomposition: The paper provides a solid theoretical foundation, proving that the total error comes from three distinct sources:
  • Approximation Error: Can the neural network actually represent the solution? (Yes, based on Sobolev space theorems).
  • Sampling Error: The noise introduced by Monte-Carlo integration.
  • Training Error: How close did the optimizer get to the actual saddle point?
  • Stability via Reformulation: By using the inf-sup formulation, the method avoids the "higher-order" issues of least-squares PINNs, making it more robust for problems with lower regularity.

Key Results

  • High-Dimensional Accuracy: In 5D Poisson problems, the model achieved a point-wise error of less than 0.01 and a relative $L^2$ error of approximately 0.6% after 20,000 iterations.
  • Robustness to Geometry: The solver successfully handled the L-shaped domain, which is notoriously difficult for many solvers due to the re-entrant corner.
  • Nonlinear Success: For nonlinear elliptic PDEs, the relative error consistently reached the ~1% range, demonstrating that the adversarial approach scales well to complex physics.
  • Convergence: Experiments confirmed that increasing the number of sampling points and the width/depth of the networks consistently reduced the error, aligning with the theoretical bounds.

Limitations and Future Work

  • Optimizer Sensitivity: Like many minimax problems (similar to GANs), the choice of optimizer and learning rate schedule is critical and can be problem-dependent.
  • Posterior vs. Prior: The current error bounds are "posterior" (calculated after the fact). The authors note that deriving "prior" estimates would be a significant theoretical step forward.
  • Time-Dependency: This paper focuses on static (elliptic) PDEs. Extending this adversarial framework to time-dependent (parabolic or hyperbolic) equations is a logical next step.
  • Efficiency: While accurate, minimax training can be computationally more expensive than simple minimization; further work is needed to optimize the training speed.

Full Paper Citation: Huo, X., & Liu, H. (2024). Inf-Sup neural networks for high-dimensional elliptic PDE problems. arXiv:2401.17233v2 [math.NA].

Summary updated

Abstract

Solving high dimensional partial differential equations (PDEs) has historically posed a considerable challenge when utilizing conventional numerical methods, such as those involving domain meshes. Recent advancements in the field have seen the emergence of neural PDE solvers, leveraging deep networks to effectively tackle high dimensional PDE problems. This study introduces Inf-SupNet, a model-based unsupervised learning approach designed to acquire solutions for a specific category of elliptic PDEs. The fundamental concept behind Inf-SupNet involves incorporating the inf-sup formulation of the underlying PDE into the loss function. The analysis reveals that the global solution error can be bounded by the sum of three distinct errors: the numerical integration error, the duality gap of the loss function (training error), and the neural network approximation error for functions within Sobolev spaces. To validate the efficacy of the proposed method, numerical experiments conducted in high dimensions demonstrate its stability and accuracy across various boundary conditions, as well as for both semi-linear and nonlinear PDEs.
BibTeX
@article{huo2024inf,
  author = {Xiaokai Huo and Hailiang Liu},
  title = {{Inf-Sup neural networks for high-dimensional elliptic PDE problems}},
  journal = {Journal of Computational Physics},
  year = {2024},
}
Figure