Transfer Operator Learning with Fusion Frame

Haoyang Jiang, Yongzhi QuarXiv 2024 · DOI · arXiv

Abstract

Operator learning models like DeepONet and Fourier Neural Operators (FNO) have transformed the numerical solution of partial differential equations (PDEs). However, these models often struggle with "brittleness" when applied to target domains with different physical parameters than the training data. This paper presents FF-POD-DeepONet, a framework that integrates fusion frame theory and Fourier Feature Networks (FFNs) to enhance transferability. By utilizing redundant mathematical subspaces, the model achieves superior generalization and data efficiency in new physical scenarios.

Introduction

Partial Differential Equations (PDEs) are fundamental to modeling complex physical systems, describing phenomena from fluid dynamics to structural mechanics. While neural operators have shown promise in learning mappings between function spaces, their performance often degrades significantly when encountering shifts in domain parameters, such as changes in fluid viscosity or material properties. This "transfer learning" challenge typically necessitates large amounts of new data or extensive retraining, which is often impractical in real-world engineering.

To address this, we introduce FF-POD-DeepONet. This architecture leverages the redundancy of fusion frames to create a more robust representation of the underlying physics, allowing for high-accuracy predictions even with limited data from the target domain.

Methodology: FF-POD-DeepONet Architecture

The proposed FF-POD-DeepONet framework combines three key components: Fourier Feature Networks (FFNs), Fusion Frame Theory, and Proper Orthogonal Decomposition (POD).

The architecture of FF-POD-DeepONet utilizes parallel branches and fusion frames to represent complex operator mappings.
  1. Fourier Feature Networks (FFNs): FFNs map input data into a high-dimensional spectral space, enabling the network to capture high-frequency patterns and intrinsic structures often missed by standard architectures.
  2. Fusion Frame Theory: Unlike traditional orthogonal bases, fusion frames allow for redundant, weighted subspaces. This redundancy provides flexibility and robustness against shifts in the underlying physical parameters, making the model less sensitive to domain shifts.
  3. Proper Orthogonal Decomposition (POD): Integrated within the DeepONet structure, POD identifies the most energetic modes of the functions, facilitating efficient dimensionality reduction within each fusion frame subspace.

The workflow involves partitioning the input space into multiple subspaces via FFNs, extracting features through POD, and processing these through parallel neural network branches. The final solution is obtained by fusing the outputs of these branches.

Experimental Evaluation

The framework was evaluated on three benchmark PDE problems: Darcy Flow, Burgers’ Equation, and an Elasticity model. These tests focused on both source domain performance and the ability to transfer to new domains with different parameters.

Darcy Flow and Transfer Performance

In modeling steady-state flow through porous media, the model was tested against shifts in Gaussian random field parameters and forcing terms. On the source dataset, FF-POD-DeepONet achieved a Mean Squared Error (MSE) of 0.0589, outperforming FNO (0.0671) and standard DeepONet (0.0803).

A key strength of the FF-POD-DeepONet is its performance in "few-shot" transfer scenarios. When adapting to a new target domain, the model maintains high accuracy with significantly fewer samples than competing methods.

MSE comparison for Darcy Flow transfer tasks across different sample sizes, highlighting the superior data efficiency of FF-POD-DeepONet.

As shown in the results for Darcy Flow transfer, FF-POD-DeepONet consistently yields the lowest MSE across all sample sizes, particularly excelling in the low-data regime (e.g., 20 samples), where it achieved an MSE of 0.2074 compared to 0.3025 for POD-DeepONet.

Discussion and Conclusion

The success of FF-POD-DeepONet highlights the value of mathematical redundancy in neural operator design. By capturing overlapping features through fusion frames, the model becomes less sensitive to variations in the target domain. Furthermore, the integration of FFNs allows for better sensitivity to the fine-grained physics essential for accurate transfer.

While the current results are promising, future work will explore automated hyperparameter tuning for subspace selection and scaling the framework to more complex, multi-scale physical problems beyond the 1D and 2D benchmarks presented here.

Reference: Jiang, H., & Qu, Y. (2024). Transfer Operator Learning with Fusion Frame. arXiv:2408.10458v1 [cs.LG].

Summary updated

Abstract

The challenge of applying learned knowledge from one domain to solve problems in another related but distinct domain, known as transfer learning, is fundamental in operator learning models that solve Partial Differential Equations (PDEs). These current models often struggle with generalization across different tasks and datasets, limiting their applicability in diverse scientific and engineering disciplines. This work presents a novel framework that enhances the transfer learning capabilities of operator learning models for solving Partial Differential Equations (PDEs) through the integration of fusion frame theory with the Proper Orthogonal Decomposition (POD)-enhanced Deep Operator Network (DeepONet). We introduce an innovative architecture that combines fusion frames with POD-DeepONet, demonstrating superior performance across various PDEs in our experimental analysis. Our framework addresses the critical challenge of transfer learning in operator learning models, paving the way for adaptable and efficient solutions across a wide range of scientific and engineering applications.
BibTeX
@misc{jiang2024transfer,
  author = {Haoyang Jiang and Yongzhi Qu},
  title = {{Transfer Operator Learning with Fusion Frame}},
  year = {2024},
  eprint = {2408.10458},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.10458},
}
Figure