Abstract
Deep operator network (DeepONet) has demonstrated great success in various learning tasks, including learning solution operators of partial differential equations. In particular, it provides an efficient approach to predict the evolution equations in a finite time horizon. Nevertheless, the vanilla DeepONet suffers from the issue of stability degradation in the long-time prediction. This paper proposes a transfer-learning aided DeepONet to enhance the stability. Our idea is to use transfer learning to sequentially update the DeepONets as the surrogates for propagators learned in different time frames. The evolving DeepONets can better track the varying complexities of the evolution equations, while only need to be updated by efficient training of a tiny fraction of the operator networks. Through systematic experiments, we show that the proposed method not only improves the long-time accuracy of DeepONet while maintaining similar computational cost but also substantially reduces the sample size of the training set.
Introduction
Deep Operator Networks (DeepONets) are a powerful class of neural networks designed to learn operators—the mathematical mappings between functions. In physical systems, this involves learning how a state evolves over time. While DeepONets excel at short-term predictions, they struggle with long-term forecasting due to two primary issues:
- Error Accumulation: Small approximation errors at each time step compound recursively, leading to explosive instabilities or non-physical results over long horizons.
- Varying Complexity: As physical systems evolve (e.g., fluid turbulence or phase separation), the underlying function space changes. A static network trained on initial conditions may lack the expressive capacity to describe the system's state much later in time.
Methodology: TL-DeepONet
The authors introduce TL-DeepONet, which combines the efficiency of DeepONets with the adaptability of transfer learning. The architecture consists of a Branch net (processing input functions) and a Trunk net (processing spatial coordinates).
The core strategy involves a two-step process:
- Pre-training: A physics-informed DeepONet is trained to learn the propagator (the mapping from $t$ to $t + \Delta t$).
- Sequential Transfer Learning: During prediction, the majority of the network is frozen. Only the weights of the final hidden layer of the branch net are updated at each time step. This re-training is mathematically equivalent to a small least-squares problem, making it computationally efficient ($O(q^2N)$ complexity) while allowing the network to "tune" its basis functions to the current state.
Key Results and Insights
The method was evaluated on several challenging partial differential equations (PDEs), including Reaction-Diffusion, Allen-Cahn, and 2D Navier-Stokes equations.
Long-Term Stability and Accuracy
TL-DeepONet demonstrates a massive gain in stability. In 1D Allen-Cahn experiments, it reduced relative $L^2$ error by two orders of magnitude compared to vanilla DeepONets and Fourier Neural Operators (FNO). For Navier-Stokes equations, while standard models failed as errors climbed toward 100%, TL-DeepONet maintained errors below 1-9% across various viscosities.
Data Efficiency and Theoretical Grounding
One of the most significant findings is data efficiency: a TL-DeepONet trained on 1,000 samples often outperformed a vanilla model trained on 10,000 samples. Theoretically, the authors prove (Theorem 1) that for non-expansive physical operators, the TL-DeepONet error grows linearly at worst, rather than exponentially. This frames the model as a "dynamic Galerkin method" that learns and updates optimal basis functions dynamically.
Conclusion
TL-DeepONet effectively solves the problem of error accumulation in neural PDE solvers. While the sequential update makes prediction slightly slower than a single forward pass, the gains in stability and accuracy are substantial. Future work will focus on more complex 3D geometries and higher-order time-stepping schemes.
Summary updated