Momentum-Accelerated Richardson(m) and Their Multilevel Neural Solvers

Zhen Wang, Yun Liu, Chen Cui, Shi ShuarXiv 2024 · DOI · arXiv

Abstract

Solving large-scale linear systems arising from the discretization of partial differential equations (PDEs) remains a significant computational challenge. This paper introduces a new class of neural solvers that leverage meta-networks to predict optimal weights for long-step Richardson iterations. By combining these learned weights with Nesterov momentum and multilevel frameworks, the proposed methods achieve up to a 20x speedup over standard neural solvers. Our approach demonstrates exceptional robustness in solving complex PDEs, such as anisotropic diffusion and high-wavenumber Helmholtz equations, while maintaining stability across varying grid scales.

Introduction

The primary bottleneck in scientific computing is often the solution of large-scale linear systems. Traditional iterative methods, such as Richardson or Jacobi, are straightforward to implement but suffer from slow convergence rates. Accelerating these methods typically requires precise knowledge of the system's spectral properties (e.g., eigenvalue bounds), which is computationally expensive to obtain. Furthermore, problems involving anisotropy or high-frequency oscillations often require hand-tuned parameters that fail to generalize across different physical constants or grid resolutions. This work proposes a hierarchical neural approach to automate parameter selection and accelerate convergence through momentum-based techniques.

Methodology

The proposed framework builds upon the Richardson(m) iteration by integrating deep learning and classical optimization theory.

Richardson(m) Neural Solver and Meta-Networks

Instead of a single fixed weight, the Richardson(m) solver utilizes a sequence of $m$ weights. We employ a "Meta-network"—a simple fully connected network—to map PDE physical parameters ($\mu$) directly to the optimal sequence of weights ($\omega$). This eliminates the need for manual eigenvalue analysis.

Architecture of the Meta-network mapping PDE parameters to optimal iteration weights.

Momentum and NAG-Extension

To further accelerate convergence, we incorporate Nesterov Accelerated Gradient (NAG) techniques. We introduce a "NAG-extension" that provides enhanced stability and faster convergence compared to standard NAG by adding a correction factor to the momentum term. The relationship between these optimization methods and their iterative counterparts is summarized in the table below.

Comparison of optimization methods (left) and their corresponding iterative Richardson formulations (right).

Multilevel Integration

To ensure mesh independence, the neural Richardson solver is integrated into advanced multilevel architectures, such as the Fourier Neural Solver (FNS) for anisotropic equations and the Wave-ADR Neural Solver (WANS) for the Helmholtz equation. This hybrid approach leverages the spectral bias of neural networks to learn low-frequency patterns while using Richardson iterations to smooth high-frequency errors.

Experimental Results

The methods were evaluated on 2D anisotropic diffusion and Helmholtz equations across grid scales ranging from $32 \times 32$ up to $4096 \times 4096$.

  • Performance Gains: The NAGex-Richardson(m)-NS with SSOR preconditioning achieved a 20x speedup in iteration counts compared to the baseline Richardson(m)-NS.
  • Mesh Independence: While single-level methods saw iteration counts explode with grid size (e.g., from 27 to 977 iterations), the multilevel Richardson(10)-FNS remained stable, requiring only 83 iterations at $N=512$.
  • Comparison to Classical Methods: The neural solvers matched or outperformed the classical Chebyshev semi-iterative method without requiring explicit eigenvalue computation.

Conclusion

This work demonstrates that meta-learning can effectively bridge the gap between deep learning and classical numerical analysis. By learning the functional relationship between a PDE's parameters and its optimal numerical weights, we create solvers that are both fast and robust. Future research will focus on extending these meta-networks to 3D systems and more complex advection-diffusion-reaction equations.

Summary updated

BibTeX
@misc{wang2024momentum,
  author = {Zhen Wang and Yun Liu and Chen Cui and Shi Shu},
  title = {{Momentum-Accelerated Richardson(m) and Their Multilevel Neural Solvers}},
  year = {2024},
  eprint = {2412.08076},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08076},
}
Figure