The parametric complexity of operator learning

Samuel Lanthaler, Andrew M. StuartIMA 2025 · DOI

TL;DR

This paper introduces the "curse of parametric complexity" for neural operator learning, showing that for general operators defined only by smoothness, achieving high accuracy requires exponentially complex models. However, they demonstrate this curse can be overcome for Hamilton-Jacobi (HJ) equations by leveraging their underlying structure, leading to a novel HJ-Net architecture with provably better complexity.

What Problem Does This Paper Address?

Operator learning aims to use neural networks to approximate functions that map between spaces of functions (e.g., solving differential equations). A key challenge is the "curse of dimensionality" (CoD), where approximating functions in high-dimensional spaces becomes computationally intractable. In operator learning, this translates to a "curse of parametric complexity": even for relatively smooth operators, the number of parameters needed for a neural operator to achieve a certain accuracy can grow exponentially with the inverse of that accuracy. This paper investigates this curse and proposes a method to overcome it for a specific class of problems.

Methods

The paper's core contributions are:

  1. Formalizing the "Curse of Parametric Complexity": The authors prove a theoretical result showing that for operators characterized solely by their $C^r$ or Lipschitz regularity, learning them with neural operators leads to an exponential lower bound on the number of parameters required for a given accuracy. This is presented as an infinite-dimensional analogue of the traditional CoD.
  2. Introducing HJ-Net: To overcome this curse for Hamilton-Jacobi (HJ) equations, they propose a novel neural operator architecture called HJ-Net. This architecture leverages the specific structure of HJ equations, particularly the method of characteristics, to learn the solution operator.
  3. Complexity Analysis: They derive error and complexity estimates for HJ-Net, demonstrating that it can indeed "beat" the curse of parametric complexity for HJ equations, achieving algebraic complexity bounds instead of exponential ones.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • The "Curse of Parametric Complexity" is Real: The paper rigorously establishes that for general operators, simply relying on smoothness ($C^r$ or Lipschitz regularity) is insufficient to avoid exponential parameter growth for high accuracy in operator learning. This is a fundamental limitation.
  • Structure is Key to Overcoming the Curse: The authors show that by exploiting specific mathematical structures within an operator (like the method of characteristics for HJ equations), one can design neural operators that bypass this curse. This highlights the importance of domain knowledge in designing effective neural operators.
  • HJ-Net: A Principled Approach for HJ Equations: The proposed HJ-Net architecture is not just a generic neural network but is carefully designed to incorporate the underlying physics of HJ equations, leading to provable theoretical advantages.

Key Results

  • General Curse: For operators with only $C^r$ or Lipschitz regularity, achieving accuracy $\epsilon$ requires model complexity that grows at least as $\exp(c\epsilon^{-1/(r+1+d)})$ (where $d$ is related to the input space dimension and $c$ is a constant). This applies to architectures like PCA-Net, DeepONet, and FNO.
  • HJ-Net Success: For HJ equations, the HJ-Net architecture achieves approximation accuracy $\epsilon$ with complexity bounds that grow only algebraically with $\epsilon^{-1}$, specifically $N < C\epsilon^{-d/r}$ for the number of encoding points and $\text{size}(\Psi_t) < C\epsilon^{-(2d+1)/r} \log(\epsilon^{-1})$ for the neural network size. This is a significant improvement over the general curse.

Limitations and Future Work

  • Classical Solutions: The HJ-Net approach relies on the existence of classical solutions to the HJ equation, which can develop singularities in finite time. The authors acknowledge this limitation and suggest future work on extending the methodology to viscosity solutions and generalized characteristics.
  • Fixed Time Interval: The analysis is primarily for short time intervals where classical solutions exist. Extending this to longer or infinite time horizons is a direction for future research.
  • Generalization: While HJ-Net overcomes the curse for HJ equations, finding a general characterization of operators that allow for efficient approximation remains an open challenge.
  • Iterated Operators: The paper focuses on approximating a single operator application. Extending the framework to iteratively apply learned operators is mentioned as an interesting future direction.
  • Symplectic Neural Networks: The authors suggest exploring symplectic neural networks for learning the Hamiltonian flow, noting that their analysis is not yet developed for such architectures.

Summary updated

Abstract

Abstract Neural operator architectures employ neural networks to approximate operators mapping between Banach spaces of functions; they may be used to accelerate model evaluations via emulation, or to discover models from data. Consequently, the methodology has received increasing attention over recent years, giving rise to the rapidly growing field of operator learning. The first contribution of this paper is to prove that for general classes of operators, which are characterized only by their $C^{r}$- or Lipschitz-regularity, operator learning suffers from a ‘curse of parametric complexity’, which is an infinite-dimensional analogue of the well-known curse of dimensionality encountered in high-dimensional approximation problems. The result is applicable to a wide variety of existing neural operators, including PCA-Net, DeepONet and the Fourier neural operator. The second contribution of the paper is to prove that this general curse can be overcome for solution operators defined by the Hamilton–Jacobi (HJ) equation; this is achieved by leveraging additional structure in the underlying solution operator, going beyond regularity. To this end a novel neural operator architecture is introduced, termed HJ-Net, which explicitly takes into account characteristic information of the underlying Hamiltonian system. Error and complexity estimates are derived for HJ-Net, which show that this architecture can provably beat the curse of parametric complexity related to the infinite-dimensional input and output function spaces.
BibTeX
@inproceedings{lanthaler2025parametric,
  author = {Samuel Lanthaler and Andrew M. Stuart},
  title = {{The parametric complexity of operator learning}},
  booktitle = {IMA},
  year = {2025},
  doi = {10.1093/imanum/draf028},
}
Figure