Abstract
Machine learning has witnessed substantial growth, leading to the development of advanced artificial intelligence models crafted to address a wide range of real-world challenges. Nevertheless, the creation of custom models for each new task remains a resource-intensive undertaking. In this study, we introduce the Neural Combinatorial Wavelet Neural Operator (NCWNO) as a foundational model for scientific computing. Designed to excel in learning from a diverse spectrum of physics, the NCWNO adapts to solution operators associated with parametric partial differential equations (PDEs) through a gated structure employing local wavelet experts. This combination enables rapid adaptation to new challenges while maintaining robustness against catastrophic forgetting. Our results demonstrate that the NCWNO outperforms task-specific baselines with minimal fine-tuning, facilitating effective knowledge transfer across dissimilar tasks.
Introduction
In traditional scientific computing, solving a new parametric partial differential equation (PDE)—such as fluid flow in a novel geometry or heat transfer with varying materials—typically requires training a model from scratch. While modern Neural Operators like FNO or DeepONet have made strides, they remain largely task-specific. Sequential training on multiple physical systems often leads to "catastrophic forgetting," where the model loses its ability to solve previous tasks as it learns new ones.
This research introduces the Neural Combinatorial Wavelet Neural Operator (NCWNO), a framework designed to learn the underlying "language" of physics across diverse PDEs. By establishing a modular library of physical features, the NCWNO serves as a foundation model that can simultaneously master multiple systems and rapidly generalize to unseen physics.
The NCWNO Architecture
The core innovation of the NCWNO lies in its modular, gated architecture, which moves away from static kernels toward a dynamic ensemble approach.
Key Components:
- Expert Wavelet Kernel Integral Blocks: The model utilizes an ensemble of "local wavelet experts." Each expert employs different wavelet basis functions to capture specific features of the solution space, such as sharp transitions or global trends.
- Gating Mechanism: A task-specific gating network processes input parameters and a PDE label to calculate mixing probabilities for the experts.
- Channel-level Ensembling: Unlike standard Mixture-of-Experts (MoE) models that select a single best expert, NCWNO performs a weighted average at the channel level. This allows the model to combine different physical behaviors—such as diffusion and advection—to solve complex, multi-physics problems.
Combinatorial Transfer and Knowledge Retention
To adapt to a new PDE, the NCWNO employs a "combinatorial transfer" strategy. The expert kernels remain frozen, and only the parameters of the gating network are fine-tuned. This treats adaptation as a retrieval problem: finding the optimal combination of existing physical skills rather than learning from scratch. This approach ensures zero forgetting of previous tasks while facilitating positive transfer, where features learned from one system improve performance on another.
Experimental Evaluation and Results
The NCWNO was evaluated on a broad range of benchmarks, including 1D and 2D time-dependent PDEs (e.g., Allen-Cahn, Burgers, Navier-Stokes), complex geometries like Darcy flow in notched domains, and chaotic systems like the Kuramoto-Sivashinsky equation.
Key Findings:
- Efficiency: Adapting to new tasks requires 3x fewer training epochs and 2x less data compared to training from scratch.
- Accuracy: NCWNO consistently outperformed task-specific baselines (DeepONet, FNO, and WNO), particularly in chaotic and complex geometry scenarios.
- Robustness: In sequential learning tests, the model maintained near-perfect accuracy on initial tasks even after learning several subsequent, dissimilar physical systems.
Conclusion
The NCWNO represents a significant step toward a universal solver for scientific computing. By leveraging wavelet-based local experts and a combinatorial gating strategy, it achieves the flexibility of a foundation model while remaining resource-efficient. Future work will focus on removing the requirement for explicit task labels and integrating physics-informed loss functions to explicitly enforce conservation laws during training.
Summary updated