HT-Net: Hierarchical Transformer based Operator Learning Model for Multiscale PDEs

Xinliang Liu, Bo Xu, Lei Zhang

TL;DR

This paper introduces HANO, a novel neural operator architecture that tackles the "spectral bias" problem in learning multiscale Partial Differential Equations (PDEs). By employing a hierarchical attention mechanism inspired by hierarchical matrices, HANO effectively captures high-frequency components, leading to superior accuracy and robustness compared to existing methods on challenging multiscale problems.

What Problem Does This Paper Address?

Many real-world phenomena, such as fluid turbulence and reservoir modeling, are governed by multiscale Partial Differential Equations (PDEs). Solving these PDEs numerically is computationally expensive, especially when the problem involves features at vastly different scales. Neural operators offer a promising data-driven approach to learn the mapping from PDE parameters to solutions, but existing methods suffer from "spectral bias." This means they tend to learn low-frequency components of the solution much faster than high-frequency ones, hindering their ability to accurately capture fine-scale details and complex oscillatory behaviors inherent in multiscale PDEs. This paper aims to bridge this gap by developing a neural operator that can effectively learn these high-frequency components.

Methods

The core innovation of this paper is the Hierarchical Attention Neural Operator (HANO). HANO is inspired by the hierarchical matrix approach, which is known for its efficiency in handling multiscale problems. The key components of HANO are:

  • Hierarchical Attention: Instead of a global attention mechanism, HANO employs a hierarchical structure. This involves a "V-cycle" of operations:
  • Reduce: Aggregates features from finer levels to coarser levels.
  • Multilevel Local Attention: Applies attention mechanisms locally within each hierarchical level, mirroring the localized nature of convolutions but with the flexibility of attention.
  • Decompose&mix: Reverses the reduce operation, propagating information back to finer levels. This hierarchical approach allows for a controllable, near-linear computational cost ($O(N)$) compared to the quadratic cost of standard self-attention ($O(N^2)$).

  • Empirical H¹ Loss Function: To further combat spectral bias and enhance the learning of high-frequency components, the authors incorporate an $H^1$ loss function. This loss function places a greater emphasis on high-frequency components compared to the standard $L^2$ loss.

  • Nested Feature Computation: HANO enables nested feature computation across a hierarchy of levels, allowing for a more nuanced encoding and decoding of the multiscale solution space.

Datasets

The paper uses several datasets for training and evaluation:

  • Porous Media Benchmark: A standard benchmark for multiscale elliptic PDEs, using a two-phase coefficient generated from a log-normal random field. This includes both "Darcy smooth" and "Darcy rough" variants to test the model's robustness to coefficient complexity.
  • Multiscale Trigonometric Coefficient Benchmark: Another benchmark for multiscale elliptic PDEs, featuring coefficients with higher contrast and non-separable scales.
  • Navier-Stokes Equation (NSE) Dataset: A benchmark for time-dependent PDEs, used to test the model's ability to capture turbulent flow dynamics.
  • Helmholtz Equation: A benchmark for acoustic wave phenomena in heterogeneous media, which is computationally expensive and challenging for capturing high-frequency components.

The authors also mention that the code and datasets are available at https://github.com/xlliu2017/HANO.

Key Insights and Ideas

  • Spectral Bias is a Major Hurdle for Multiscale PDEs: The paper provides strong empirical evidence that existing neural operators struggle significantly with learning high-frequency components in multiscale PDEs, leading to inaccurate predictions of fine-scale features.
  • Hierarchical Structure Mimics Multigrid Methods: The V-cycle approach in HANO, with its reduce and decompose operations, draws inspiration from efficient multigrid methods used in classical numerical solvers. This suggests a promising avenue for combining the strengths of deep learning and traditional numerical techniques.
  • Local Attention is Key for Efficiency and Effectiveness: By restricting attention to local neighborhoods within each hierarchical level, HANO achieves near-linear complexity while still capturing relevant interactions, avoiding the computational bottleneck of global attention.
  • H¹ Loss is Crucial for High-Frequency Learning: The empirical $H^1$ loss function demonstrably improves the model's ability to learn high-frequency components, leading to better accuracy and generalization, especially for oscillatory solutions.
  • HANO's Adaptability to Input: The hierarchical attention mechanism allows HANO to adapt its kernel to specific input instances, a key advantage over methods that rely on fixed kernel integrations.

Key Results

  • Superior Accuracy: HANO consistently outperforms state-of-the-art (SOTA) neural operators (e.g., FNO, UNet, U-NO, MWT, GT, SWIN) across various multiscale PDE benchmarks, particularly in terms of relative $L^2$ and $H^1$ errors. For instance, in the Darcy rough case, HANO achieves significantly lower errors than other methods.
  • Mitigation of Spectral Bias: Figure 7 clearly shows that HANO's training error dynamics decay faster for higher frequencies and are more uniform compared to other methods, indicating a successful mitigation of spectral bias.
  • Robustness to Complexity: HANO demonstrates enhanced accuracy and robustness when dealing with more complex and rough coefficients in multiscale problems.
  • Efficiency: Despite its enhanced capabilities, HANO maintains a competitive computational cost, achieving near-linear complexity. Table 5 shows that HANO has a lower memory usage than some other models like MWT and GT, even while achieving higher accuracy.
  • Discretization Invariance: HANO shows promising discretization invariance, allowing models trained on lower-resolution data to be evaluated at higher resolutions with minimal performance degradation, outperforming FNO in this regard.

Limitations and Future Work

  • Grid Dependency: The current implementation of HANO requires a regular grid. The authors suggest extending it to data clouds and graph neural networks to exploit its hierarchical representation more broadly.
  • Aliasing Errors: While HANO can achieve discretization invariance through interpolation, both simple interpolation and Fourier interpolation (used by FNO) can suffer from aliasing errors in the frequency domain. Future work could explore operator-adaptive sampling and interpolation techniques to better balance discretization invariance and model accuracy.
  • Further Exploration of Attention Mechanisms: The paper hints at the potential for further improvements by exploring more sophisticated attention mechanisms and their integration with hierarchical structures.

Summary updated

Abstract

Neural operators have emerged as a powerful tool for learning the mapping between infinite-dimensional parameter and solution spaces of partial differential equations (PDEs). In this work, we focus on multiscale PDEs that have important applications such as reservoir modeling and turbulence prediction. We demonstrate that for such PDEs, the spectral bias towards low-frequency components presents a significant challenge for existing neural operators. To address this challenge, we propose a hierarchical attention neural operator (HANO) inspired by the hierarchical matrix approach. HANO features a scale-adaptive interaction range and self-attentions over a hierarchy of levels, enabling nested feature computation with controllable linear cost and encoding/decoding of multiscale solution space. We also incorporate an empirical $H^1$ loss function to enhance the learning of high-frequency components. Our numerical experiments demonstrate that HANO outperforms state-of-the-art (SOTA) methods for representative multiscale problems.
BibTeX
@inproceedings{liu2023ht,
  author = {Xinliang Liu and Bo Xu and Lei Zhang},
  title = {{HT-Net: Hierarchical Transformer based Operator Learning Model for Multiscale PDEs}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=UY5zS0OsK2e},
}
Figure