Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equations

Wuzhe Xu, Yulong Lu, Li Wang

Abstract

Deep operator network (DeepONet) has demonstrated great success in various learning tasks, including learning solution operators of partial differential equations. In particular, it provides an efficient approach to predict the evolution equations in a finite time horizon. Nevertheless, the vanilla DeepONet suffers from the issue of stability degradation in the long-time prediction. This paper proposes a transfer-learning aided DeepONet to enhance the stability. Our idea is to use transfer learning to sequentially update the DeepONets as the surrogates for propagators learned in different time frames. The evolving DeepONets can better track the varying complexities of the evolution equations, while only need to be updated by efficient training of a tiny fraction of the operator networks. Through systematic experiments, we show that the proposed method not only improves the long-time accuracy of DeepONet while maintaining similar computational cost but also substantially reduces the sample size of the training set.

Introduction

Deep Operator Networks (DeepONets) are a powerful class of neural networks designed to learn operators—the mathematical mappings between functions. In physical systems, this involves learning how a state evolves over time. While DeepONets excel at short-term predictions, they struggle with long-term forecasting due to two primary issues:

  1. Error Accumulation: Small approximation errors at each time step compound recursively, leading to explosive instabilities or non-physical results over long horizons.
  2. Varying Complexity: As physical systems evolve (e.g., fluid turbulence or phase separation), the underlying function space changes. A static network trained on initial conditions may lack the expressive capacity to describe the system's state much later in time.

Methodology: TL-DeepONet

The authors introduce TL-DeepONet, which combines the efficiency of DeepONets with the adaptability of transfer learning. The architecture consists of a Branch net (processing input functions) and a Trunk net (processing spatial coordinates).

Architecture of the TL-DeepONet showing the branch and trunk networks and the sequential update mechanism.

The core strategy involves a two-step process:

  • Pre-training: A physics-informed DeepONet is trained to learn the propagator (the mapping from $t$ to $t + \Delta t$).
  • Sequential Transfer Learning: During prediction, the majority of the network is frozen. Only the weights of the final hidden layer of the branch net are updated at each time step. This re-training is mathematically equivalent to a small least-squares problem, making it computationally efficient ($O(q^2N)$ complexity) while allowing the network to "tune" its basis functions to the current state.

Key Results and Insights

The method was evaluated on several challenging partial differential equations (PDEs), including Reaction-Diffusion, Allen-Cahn, and 2D Navier-Stokes equations.

Long-Term Stability and Accuracy

TL-DeepONet demonstrates a massive gain in stability. In 1D Allen-Cahn experiments, it reduced relative $L^2$ error by two orders of magnitude compared to vanilla DeepONets and Fourier Neural Operators (FNO). For Navier-Stokes equations, while standard models failed as errors climbed toward 100%, TL-DeepONet maintained errors below 1-9% across various viscosities.

Comparison of relative L2 error over time for various models, demonstrating the superior stability of TL-DeepONet.

Data Efficiency and Theoretical Grounding

One of the most significant findings is data efficiency: a TL-DeepONet trained on 1,000 samples often outperformed a vanilla model trained on 10,000 samples. Theoretically, the authors prove (Theorem 1) that for non-expansive physical operators, the TL-DeepONet error grows linearly at worst, rather than exponentially. This frames the model as a "dynamic Galerkin method" that learns and updates optimal basis functions dynamically.

Conclusion

TL-DeepONet effectively solves the problem of error accumulation in neural PDE solvers. While the sequential update makes prediction slightly slower than a single forward pass, the gains in stability and accuracy are substantial. Future work will focus on more complex 3D geometries and higher-order time-stepping schemes.

Summary updated

Abstract

Deep operator network (DeepONet) has demonstrated great success in various learning tasks, including learning solution operators of partial differential equations. In particular, it provides an efficient approach to predict the evolution equations in a finite time horizon. Nevertheless, the vanilla DeepONet suffers from the issue of stability degradation in the long-time prediction. This paper proposes a {\em transfer-learning} aided DeepONet to enhance the stability. Our idea is to use transfer learning to sequentially update the DeepONets as the surrogates for propagators learned in different time frames. The evolving DeepONets can better track the varying complexities of the evolution equations, while only need to be updated by efficient training of a tiny fraction of the operator networks. Through systematic experiments, we show that the proposed method not only improves the long-time accuracy of DeepONet while maintaining similar computational cost but also substantially reduces the sample size of the training set.
BibTeX
@inproceedings{xu2023transfer,
  author = {Wuzhe Xu and Yulong Lu and Li Wang},
  title = {{Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26262},
}
Figure