A foundational neural operator that continuously learns without forgetting

Tapas Tripura, Souvik ChakrabortyarXiv 2023 · OpenReview

Abstract

Machine learning has witnessed substantial growth, leading to the development of advanced artificial intelligence models crafted to address a wide range of real-world challenges. Nevertheless, the creation of custom models for each new task remains a resource-intensive undertaking. In this study, we introduce the Neural Combinatorial Wavelet Neural Operator (NCWNO) as a foundational model for scientific computing. Designed to excel in learning from a diverse spectrum of physics, the NCWNO adapts to solution operators associated with parametric partial differential equations (PDEs) through a gated structure employing local wavelet experts. This combination enables rapid adaptation to new challenges while maintaining robustness against catastrophic forgetting. Our results demonstrate that the NCWNO outperforms task-specific baselines with minimal fine-tuning, facilitating effective knowledge transfer across dissimilar tasks.

Introduction

In traditional scientific computing, solving a new parametric partial differential equation (PDE)—such as fluid flow in a novel geometry or heat transfer with varying materials—typically requires training a model from scratch. While modern Neural Operators like FNO or DeepONet have made strides, they remain largely task-specific. Sequential training on multiple physical systems often leads to "catastrophic forgetting," where the model loses its ability to solve previous tasks as it learns new ones.

This research introduces the Neural Combinatorial Wavelet Neural Operator (NCWNO), a framework designed to learn the underlying "language" of physics across diverse PDEs. By establishing a modular library of physical features, the NCWNO serves as a foundation model that can simultaneously master multiple systems and rapidly generalize to unseen physics.

The NCWNO Architecture

The core innovation of the NCWNO lies in its modular, gated architecture, which moves away from static kernels toward a dynamic ensemble approach.

Overview of the NCWNO architecture, including the expert blocks, gating mechanism, and combinatorial transfer strategy.

Key Components:

  1. Expert Wavelet Kernel Integral Blocks: The model utilizes an ensemble of "local wavelet experts." Each expert employs different wavelet basis functions to capture specific features of the solution space, such as sharp transitions or global trends.
  2. Gating Mechanism: A task-specific gating network processes input parameters and a PDE label to calculate mixing probabilities for the experts.
  3. Channel-level Ensembling: Unlike standard Mixture-of-Experts (MoE) models that select a single best expert, NCWNO performs a weighted average at the channel level. This allows the model to combine different physical behaviors—such as diffusion and advection—to solve complex, multi-physics problems.

Combinatorial Transfer and Knowledge Retention

To adapt to a new PDE, the NCWNO employs a "combinatorial transfer" strategy. The expert kernels remain frozen, and only the parameters of the gating network are fine-tuned. This treats adaptation as a retrieval problem: finding the optimal combination of existing physical skills rather than learning from scratch. This approach ensures zero forgetting of previous tasks while facilitating positive transfer, where features learned from one system improve performance on another.

Experimental Evaluation and Results

The NCWNO was evaluated on a broad range of benchmarks, including 1D and 2D time-dependent PDEs (e.g., Allen-Cahn, Burgers, Navier-Stokes), complex geometries like Darcy flow in notched domains, and chaotic systems like the Kuramoto-Sivashinsky equation.

Comparison of ground truth and NCWNO predictions for five different 1D PDEs across varying initial conditions.

Key Findings:

  • Efficiency: Adapting to new tasks requires 3x fewer training epochs and 2x less data compared to training from scratch.
  • Accuracy: NCWNO consistently outperformed task-specific baselines (DeepONet, FNO, and WNO), particularly in chaotic and complex geometry scenarios.
  • Robustness: In sequential learning tests, the model maintained near-perfect accuracy on initial tasks even after learning several subsequent, dissimilar physical systems.

Conclusion

The NCWNO represents a significant step toward a universal solver for scientific computing. By leveraging wavelet-based local experts and a combinatorial gating strategy, it achieves the flexibility of a foundation model while remaining resource-efficient. Future work will focus on removing the requirement for explicit task labels and integrating physics-informed loss functions to explicitly enforce conservation laws during training.

Summary updated

Abstract

Machine learning has witnessed substantial growth, leading to the development of advanced artificial intelligence models crafted to address a wide range of real-world challenges spanning various domains, such as computer vision, natural language processing, and scientific computing. Nevertheless, the creation of custom models for each new task remains a resource-intensive undertaking, demanding considerable computational time and memory resources. In this study, we introduce the concept of the Neural Combinatorial Wavelet Neural Operator (NCWNO) as a foundational model for scientific computing. This model is specifically designed to excel in learning from a diverse spectrum of physics and continuously adapt to the solution operators associated with parametric partial differential equations (PDEs). The NCWNO leverages a gated structure that employs local wavelet experts to acquire shared features across multiple physical systems, complemented by a memory-based ensembling approach among these local wavelet experts. This combination enables rapid adaptation to new challenges. The proposed foundational model offers two key advantages: (i) it can simultaneously learn solution operators for multiple parametric PDEs, and (ii) it can swiftly generalize to new parametric PDEs with minimal fine-tuning. The proposed NCWNO is the first foundational operator learning algorithm distinguished by its (i) robustness against catastrophic forgetting, (ii) the maintenance of positive transfer for new parametric PDEs, and (iii) the facilitation of knowledge transfer across dissimilar tasks. Through an extensive set of benchmark examples, we demonstrate that the NCWNO can outperform task-specific baseline operator learning frameworks with minimal hyperparameter tuning at the prediction stage. We also show that with minimal fine-tuning, the NCWNO performs accurate combinatorial learning of new parametric PDEs.
BibTeX
@misc{tripura2023foundational,
  author = {Tapas Tripura and Souvik Chakraborty},
  title = {{A foundational neural operator that continuously learns without forgetting}},
  year = {2023},
  eprint = {2310.18885},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18885},
}
Figure