Abstract
Traditional Neural Operators for Scientific Machine Learning (SciML) often struggle with real-time applications due to fixed-length input requirements and a lack of temporal causality. This paper introduces B-LSTM-MIONet, a novel framework that integrates Long Short-Term Memory (LSTM) networks into the Multiple-Input Deep Neural Operator (MIONet) architecture. By employing an FNN-LSTM-FNN "sandwich" structure, the model handles variable-length temporal histories while maintaining causality. Furthermore, a Bayesian approach using replica exchange Stochastic Gradient Langevin Diffusion (reSGLD) provides robust uncertainty quantification. Evaluated on chaotic systems, control problems, and real-world solar data, B-LSTM-MIONet demonstrates superior accuracy, extrapolation capabilities, and reliable confidence intervals.
Introduction
Neural Operators, such as DeepONet, have revolutionized the learning of operators in physical systems, mapping input functions (like external forces) to output trajectories. However, standard architectures face significant hurdles in real-world, real-time scenarios. They typically require fixed-length input windows, which is impractical for streaming data where the available history varies. Moreover, many models lack inherent temporal causality and fail to provide uncertainty estimates for noisy sensor data.
Methodology
The proposed B-LSTM-MIONet extends the Multiple-Input Deep Neural Operator (MIONet) to address these limitations by specializing sub-networks for different aspects of the dynamical system.
The Architecture
The framework decomposes the learning task into three specialized sub-networks:
- Branch Net 1 (Current State): A Feed-Forward Network (FNN) encoding the system's state at the current time $x(t)$.
- Branch Net 2 (Temporal History): An FNN-LSTM-FNN hybrid that processes input sequences of arbitrary length, allowing the model to capture past dynamics flexibly.
- Trunk Net (Time Step): Encodes the look-ahead time $h$ to predict the future state $x(t+h)$.
Bayesian Uncertainty Quantification (UQ)
To account for sensor noise and model uncertainty, the authors utilize replica exchange Stochastic Gradient Langevin Diffusion (reSGLD). This Bayesian framework samples an ensemble of models from the posterior distribution of parameters, enabling the generation of 95% confidence intervals for all predictions, telling the user exactly how much to trust the output.
Experimental Evaluation
The model was validated across three increasingly complex scenarios:
- Lorentz 63 System: A classic autonomous, chaotic system used to test performance under high unpredictability.
- Pendulum Swing-Up: A non-autonomous control system tested with "out-of-distribution" control signals to check generalization.
- Ausgrid Solar PV Dataset: Real-world data from 300 residential solar installations in Australia, testing robustness against stochastic engineering data.
Key Results and Insights
- Superior Accuracy: LSTM-MIONet achieved a 1.29% relative error on the Lorentz system, significantly outperforming standard DeepONet (5.23%).
- Robustness to Noise: In the pendulum experiments, the Bayesian 95% confidence intervals successfully captured 100% of true values despite Gaussian noise.
- Extrapolation Capability: The model successfully predicted system behavior in time domains 120% longer than the training window with minimal accuracy loss.
- Causality by Design: The architectural separation of current state and history ensures the model respects temporal causality, a feature often missing in pure FNN-based operators.
Discussion and Conclusion
While B-LSTM-MIONet shows remarkable flexibility, its error increases linearly with the prediction step size ($h$), as the relevance of historical data diminishes over longer horizons. Future work will explore combining Recurrent Neural Networks with Fourier Neural Operators (FNOs) to enhance long-term stability. In conclusion, B-LSTM-MIONet provides a robust, real-time capable solution for modeling complex dynamical systems with inherent uncertainty quantification.
Summary updated