B-LSTM-MIONet: Bayesian LSTM-based Neural Operators for Learning the Response of Complex Dynamical Systems to Length-Variant Multiple Input Functions

Guang Lin, Zhihao Kong, Amirhossein Mollaali, Christian Moya, Na Lu, Guang Lin 0001arXiv 2023 · DOI · arXiv

Abstract

Traditional Neural Operators for Scientific Machine Learning (SciML) often struggle with real-time applications due to fixed-length input requirements and a lack of temporal causality. This paper introduces B-LSTM-MIONet, a novel framework that integrates Long Short-Term Memory (LSTM) networks into the Multiple-Input Deep Neural Operator (MIONet) architecture. By employing an FNN-LSTM-FNN "sandwich" structure, the model handles variable-length temporal histories while maintaining causality. Furthermore, a Bayesian approach using replica exchange Stochastic Gradient Langevin Diffusion (reSGLD) provides robust uncertainty quantification. Evaluated on chaotic systems, control problems, and real-world solar data, B-LSTM-MIONet demonstrates superior accuracy, extrapolation capabilities, and reliable confidence intervals.

Introduction

Neural Operators, such as DeepONet, have revolutionized the learning of operators in physical systems, mapping input functions (like external forces) to output trajectories. However, standard architectures face significant hurdles in real-world, real-time scenarios. They typically require fixed-length input windows, which is impractical for streaming data where the available history varies. Moreover, many models lack inherent temporal causality and fail to provide uncertainty estimates for noisy sensor data.

The standard DeepONet architecture learns to map an input function $u$ to an output $G(u)(t)$ at a query point $t$.

Methodology

The proposed B-LSTM-MIONet extends the Multiple-Input Deep Neural Operator (MIONet) to address these limitations by specializing sub-networks for different aspects of the dynamical system.

The Architecture

The framework decomposes the learning task into three specialized sub-networks:

  1. Branch Net 1 (Current State): A Feed-Forward Network (FNN) encoding the system's state at the current time $x(t)$.
  2. Branch Net 2 (Temporal History): An FNN-LSTM-FNN hybrid that processes input sequences of arbitrary length, allowing the model to capture past dynamics flexibly.
  3. Trunk Net (Time Step): Encodes the look-ahead time $h$ to predict the future state $x(t+h)$.
The MIONet architecture allows for multiple input functions ($v^1, ..., v^n$), providing the structural basis for the multi-branch LSTM-MIONet.

Bayesian Uncertainty Quantification (UQ)

To account for sensor noise and model uncertainty, the authors utilize replica exchange Stochastic Gradient Langevin Diffusion (reSGLD). This Bayesian framework samples an ensemble of models from the posterior distribution of parameters, enabling the generation of 95% confidence intervals for all predictions, telling the user exactly how much to trust the output.

Experimental Evaluation

The model was validated across three increasingly complex scenarios:

  • Lorentz 63 System: A classic autonomous, chaotic system used to test performance under high unpredictability.
  • Pendulum Swing-Up: A non-autonomous control system tested with "out-of-distribution" control signals to check generalization.
  • Ausgrid Solar PV Dataset: Real-world data from 300 residential solar installations in Australia, testing robustness against stochastic engineering data.

Key Results and Insights

  • Superior Accuracy: LSTM-MIONet achieved a 1.29% relative error on the Lorentz system, significantly outperforming standard DeepONet (5.23%).
  • Robustness to Noise: In the pendulum experiments, the Bayesian 95% confidence intervals successfully captured 100% of true values despite Gaussian noise.
  • Extrapolation Capability: The model successfully predicted system behavior in time domains 120% longer than the training window with minimal accuracy loss.
  • Causality by Design: The architectural separation of current state and history ensures the model respects temporal causality, a feature often missing in pure FNN-based operators.

Discussion and Conclusion

While B-LSTM-MIONet shows remarkable flexibility, its error increases linearly with the prediction step size ($h$), as the relevance of historical data diminishes over longer horizons. Future work will explore combining Recurrent Neural Networks with Fourier Neural Operators (FNOs) to enhance long-term stability. In conclusion, B-LSTM-MIONet provides a robust, real-time capable solution for modeling complex dynamical systems with inherent uncertainty quantification.

Summary updated

Abstract

Deep Operator Network (DeepONet) is a neural network framework for learning nonlinear operators such as those from ordinary differential equations (ODEs) describing complex systems. Multiple-input deep neural operators (MIONet) extended DeepONet to allow multiple input functions in different Banach spaces. MIONet offers flexibility in training dataset grid spacing, without constraints on output location. However, it requires offline inputs and cannot handle varying sequence lengths in testing datasets, limiting its real-time application in dynamic complex systems. This work redesigns MIONet, integrating Long Short Term Memory (LSTM) to learn neural operators from time-dependent data. This approach overcomes data discretization constraints and harnesses LSTM's capability with variable-length, real-time data. Factors affecting learning performance, like algorithm extrapolation ability are presented. The framework is enhanced with uncertainty quantification through a novel Bayesian method, sampling from MIONet parameter distributions. Consequently, we develop the B-LSTM-MIONet, incorporating LSTM's temporal strengths with Bayesian robustness, resulting in a more precise and reliable model for noisy datasets.
BibTeX
@misc{lin2023b,
  author = {Guang Lin and Zhihao Kong and Amirhossein Mollaali and Christian Moya and Na Lu and Guang Lin},
  title = {{B-LSTM-MIONet: Bayesian LSTM-based Neural Operators for Learning the Response of Complex Dynamical Systems to Length-Variant Multiple Input Functions}},
  year = {2023},
  eprint = {2311.16519},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.16519},
}
Figure