Fed-DeepONet: Stochastic Gradient-Based Federated Training of Deep Operator Networks

Guang Lin, Christian Moya, Guang Lin 0001Algorithms 2022 · DOI

TL;DR

This paper introduces Fed-DeepONet, a novel framework for federated training of Deep Operator Networks (DeepONets). By adapting stochastic gradient descent for a distributed setting, Fed-DeepONet enables collaborative training of complex nonlinear operators across multiple clients while preserving data privacy, addressing a key limitation of traditional centralized DeepONet training.

What Problem Does This Paper Address?

Deep Operator Networks (DeepONets) are powerful tools for learning complex, nonlinear mappings between infinite-dimensional spaces, such as the solution operators of dynamical systems. However, traditional DeepONet training requires aggregating all data onto a central server. This centralized approach poses significant challenges: it compromises data privacy, making it unsuitable for sensitive applications like digital twins, and it prevents leveraging distributed high-performance computing resources. The paper aims to bridge this gap by developing a federated learning approach for DeepONets, allowing them to be trained collaboratively across multiple clients without sharing their raw data.

Methods

The core contribution is the development of Fed-DeepONet, a federated training framework for DeepONets. The authors propose two main algorithms:

  • Fed-DeepONet (Algorithm 1): This algorithm adapts the standard federated averaging (FedAvg) approach. It involves clients performing local stochastic gradient descent updates on their DeepONet models and then averaging these updates at a central server. This is achieved by averaging first-order estimates of the DeepONet loss gradient across clients.
  • Adaptive Fed-DeepONet (Algorithm 2): To accelerate convergence, this algorithm enhances Fed-DeepONet by incorporating a moment-enhanced (adaptive) stochastic gradient-based strategy, similar to optimizers like Adam. This adaptive approach helps to speed up the training process, especially in non-convex scenarios.

Both algorithms are designed to handle potentially heterogeneous data distributions across clients.

Datasets

The paper does not introduce new benchmark datasets. Instead, it uses synthetically generated data for its experiments. The data is generated using the true operator $G$ for the pendulum system and a library of pendulums. For training, $N=10,000$ data triplets are generated and then distributed among clients. For evaluation, 100 test trajectories are used. The input functions are generated from Gaussian Random Fields (GRF) and also include out-of-distribution functions like $t$, $\sin(\pi t)$, and $t\sin(2\pi t)$.

Key Insights and Ideas

  • Federated DeepONets are feasible: The paper demonstrates that DeepONets can indeed be trained effectively in a federated setting, preserving their powerful operator learning capabilities.
  • Adaptive methods are crucial for speed: The adaptive version of Fed-DeepONet significantly outperforms the basic Fed-DeepONet in terms of training convergence speed, highlighting the importance of adaptive optimizers in federated learning for complex models.
  • Robustness to Data Heterogeneity: Fed-DeepONet shows promising robustness to various forms of data heterogeneity, including differences in input distributions and conditional distributions, which are common in real-world federated learning scenarios.
  • Privacy Preservation: By design, the federated approach inherently protects client data privacy, a critical advantage over centralized training.
  • Generalization Power: Fed-DeepONet maintains the strong generalization capabilities of centralized DeepONets, even when trained on distributed data.

Key Results

The paper presents several key quantitative and qualitative results:

  • Convergence Comparison: Figure 2 clearly shows that the Adaptive Fed-DeepONet (Algorithm 2) converges much faster than the standard Fed-DeepONet (Algorithm 1).
  • Pendulum Operator Learning: For the gravity pendulum, the trained Fed-DeepONet achieved a mean L2-relative error of 1.362% for test trajectories generated by a Gaussian Random Field.
  • Out-of-Distribution Performance: Fed-DeepONet demonstrated good generalization on out-of-distribution inputs, with L2-relative errors around 1-2% for various test cases.
  • Sensitivity to Client Configurations: Tables 1 and 2 show that Fed-DeepONet is robust to variations in the number of clients ($C$) and the fraction of available clients ($\alpha$). For instance, with 20 clients and 75% availability, the mean error was 1.154%. Even with only 25% of clients available, the mean error remained competitive at 1.495%.
  • Data Heterogeneity: Figures 5 and 6 illustrate that Fed-DeepONet (especially the adaptive version) performs well even under significant data heterogeneity, with errors generally remaining low.
  • Parametric Library Learning: In Experiment 4, Fed-DeepONet successfully learned the dynamic response of a library of pendulums with varying parameters, achieving a mean L2-relative error of 2.582% for in-distribution test inputs.

Limitations and Future Work

The authors acknowledge several limitations and propose future research directions:

  • Offline Training: Fed-DeepONet currently requires offline training and cannot assimilate data in real-time. This is a limitation inherited from batch stochastic gradient descent. Future work could explore distributed Kalman Filters for real-time adaptation.
  • Information Leakage: While data privacy is preserved, information about client processes might still be inferred from transmitted parameters. Future work could explore keeping some parameters private to mitigate this.
  • Multi-Objective Optimization: Fed-DeepONet may struggle with multi-objective loss functions or constrained problems. Future work could involve penalty methods or federated evolutionary strategies.
  • Future Applications: The authors plan to apply Fed-DeepONet to build digital twins for renewable energy resources and city-wide predictive models. They also aim to accelerate material science research and extend the framework to federated sampling for uncertainty quantification.

Summary updated

BibTeX
@article{lin2022fed,
  author = {Guang Lin and Christian Moya and Guang Lin},
  title = {{Fed-DeepONet: Stochastic Gradient-Based Federated Training of Deep Operator Networks}},
  journal = {Algorithms},
  year = {2022},
  doi = {10.3390/a15090325},
}
Figure