Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equations

Wuzhe Xu, Yulong Lu, Li Wang

Abstract

Deep Operator Networks (DeepONets) have emerged as efficient surrogate models for learning solution operators of partial differential equations (PDEs). However, recursive predictions using these models often suffer from "stability degradation," where errors accumulate over long time horizons. This paper introduces a transfer learning (TL) framework that addresses this issue by sequentially updating only the last layer of a pre-trained DeepONet using physics-informed losses. This method achieves stable, high-accuracy predictions for complex evolution equations, such as Navier-Stokes and Allen-Cahn, while requiring significantly less training data than standard approaches.

Introduction

Partial Differential Equations (PDEs) are fundamental to modeling physical systems, yet traditional numerical solvers are often computationally prohibitive for real-time applications. Deep learning models like DeepONet offer a faster alternative by learning the mapping from initial conditions to future states. Despite their potential, these models face a critical challenge: long-time instability. In recursive prediction settings, small errors at each step compound, leading to simulations that eventually diverge from physical reality. Furthermore, static networks often fail to adapt to the evolving physical complexity of dynamic systems.

Methodology

The proposed Transfer Learning (TL) aided DeepONet treats the network as an evolving operator rather than a static one. The architecture leverages the standard DeepONet structure, comprising a Branch net to encode initial conditions and a Trunk net for spatial coordinates.

Architecture of the TL-DeepONet, illustrating the Branch and Trunk networks and the weight update mechanism.

The core innovation lies in the sequential updating process:

  1. Feature Extraction: The majority of the network, including the Trunk net and the hidden layers of the Branch net, is frozen after initial training to act as fixed feature extractors.
  2. Sequential Updating: At each prediction step, only the weights ($w$) of the last hidden layer in the Branch net are re-trained.
  3. Physics-Informed Loss: Updates are driven by a physics-informed loss function based on the PDE residual, eliminating the need for labeled ground-truth data during the prediction phase.
  4. Efficiency: By restricting optimization to the final linear layer, the update becomes a computationally inexpensive least-squares problem.

Experimental Evaluation

The framework was tested on several challenging evolution equations, including the Reaction-Diffusion equation, the Allen-Cahn and Cahn-Hilliard equations (notorious for sharp transition layers), and the 2D Navier-Stokes equation. It was also evaluated on the Multiscale Linear Radiative Transfer Equation (RTE), testing performance across different physical regimes.

Key Results and Insights

The TL-DeepONet demonstrates significant improvements in both stability and data efficiency. By viewing DeepONet as a learned Galerkin method—where the Trunk net provides basis functions and the Branch net provides coefficients—the transfer learning step allows these coefficients to adapt to changing solution regularity.

Relative L2 error over time for various models. TL-DeepONet maintains stability (~10^-3) while standard DeepONet and FNO errors explode.

Key findings include:

  • Long-Term Stability: In the 1D Allen-Cahn equation, TL-DeepONet maintained a relative $L^2$ error of approximately $10^{-3}$ up to $T=50$, whereas the vanilla DeepONet's error reached $10^0$ almost immediately.
  • Fluid Dynamics: For 2D Navier-Stokes simulations, the TL approach reduced prediction errors by two orders of magnitude compared to standard models.
  • Data Efficiency: A TL-DeepONet trained on 1,000 samples outperformed a standard model trained on 10,000 samples, highlighting its robustness with limited data.

Conclusion

The transfer learning framework effectively mitigates error accumulation in long-term PDE simulations. While the method relies on the initial training data being sufficiently diverse to "prime" the frozen Trunk net with necessary spatial features, it offers a powerful computational balance between pure forward passes and full retraining. Future work may explore more advanced time-discretization schemes and applications to high-Reynolds-number turbulent flows.

Summary updated

Abstract

Deep operator network (DeepONet) has demonstrated great success in various learning tasks, including learning solution operators of partial differential equations. In particular, it provides an efficient approach to predict the evolution equations in a finite time horizon. Nevertheless, the vanilla DeepONet suffers from the issue of stability degradation in the long-time prediction. This paper proposes a {\em transfer-learning} aided DeepONet to enhance the stability. Our idea is to use transfer learning to sequentially update the DeepONets as the surrogates for propagators learned in different time frames. The evolving DeepONets can better track the varying complexities of the evolution equations, while only need to be updated by efficient training of a tiny fraction of the operator networks. Through systematic experiments, we show that the proposed method not only improves the long-time accuracy of DeepONet while maintaining similar computational cost but also substantially reduces the sample size of the training set.
BibTeX
@inproceedings{xu2023transfer,
  author = {Wuzhe Xu and Yulong Lu and Li Wang},
  title = {{Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26262},
}
Figure