Combining Machine Learning with Knowledge-Based Modeling for Scalable Forecasting and Subgrid-Scale Closure of Large, Complex, Spatiotemporal Systems

Alexander Wikner, Jaideep Pathak, Brian R. Hunt, Michelle Girvan, Troy Arcomano, Istvan Szunyogh, Andrew Pomerance, Edward OttarXiv 2020 · DOI · OpenReview

Abstract

We consider the commonly encountered situation (e.g., in weather forecasting) where the goal is to predict the time evolution of a large, spatiotemporally chaotic dynamical system when we have access to both time series data of previous system states and an imperfect model of the full system dynamics. Specifically, we attempt to utilize machine learning as the essential tool for integrating the use of past data into predictions. In order to facilitate scalability to the common scenario of interest where the spatiotemporally chaotic system is very large and complex, we propose combining two approaches: (i) a parallel machine learning prediction scheme; and (ii) a hybrid technique, for a composite prediction system composed of a knowledge-based component and a machine-learning-based component. We demonstrate that not only can this method combining (i) and (ii) be scaled to give excellent performance for very large systems, but also that the length of time series data needed to train our multiple, parallel machine learning components is dramatically less than that necessary without parallelization. Furthermore, considering cases where computational realization of the knowledge-based component does not resolve subgrid-scale processes, our scheme is able to use training data to incorporate the effect of the unresolved short-scale dynamics upon the resolved longer-scale dynamics ("subgrid-scale closure").

Introduction

Predicting the future state of large, chaotic systems—such as global weather patterns or ocean currents—remains one of the most significant challenges in computational science. These systems are characterized by the "curse of dimensionality," where a single global model may need to track upwards of $10^9$ variables.

Traditional approaches rely on knowledge-based models derived from physical equations. While powerful, these models are often "imperfect" because they cannot computationally afford to resolve tiny, subgrid-scale processes (e.g., individual cloud formations) that eventually influence large-scale dynamics. Conversely, pure machine learning (ML) models can learn directly from data but often fail to scale to millions of variables and require prohibitive amounts of training data. This paper introduces CHyPP (Combined Hybrid/Parallel Prediction), a framework designed to bridge these two paradigms.

The CHyPP Framework

The CHyPP architecture integrates three core pillars to achieve scalable, data-efficient forecasting:

  1. Reservoir Computing (RC): A type of Recurrent Neural Network where internal connections are fixed and random. Only the output layer is trained via linear regression, making it exceptionally fast and computationally inexpensive compared to deep learning alternatives.
  2. Hybridization: Rather than learning the system from scratch, the ML component is fed the output of an imperfect physics model. The ML learns to "fix" the errors and account for the subgrid-scale dynamics that the physics model misses.
  3. Parallelization: The spatial domain is divided into smaller, overlapping regions, each assigned its own reservoir computer. These units operate in parallel, with overlaps accounting for local causal interactions.
Schematic of the machine learning (ML) component in (a) training/one-step prediction mode and (b) autonomous prediction mode where outputs are fed back as inputs.

As shown in the figure above, the ML component operates in two phases. During training (a), it learns the mapping from the current state to the next. During the prediction phase (b), the system becomes autonomous, feeding its own previous predictions back into the input to forecast the system's evolution over time.

Experimental Validation

The researchers benchmarked CHyPP using two primary simulated systems:

  • Kuramoto-Sivashinsky (KS) Equation: A 1D model for spatiotemporal chaos used to test scaling as system size increases.
  • Lorenz Model III: A multiscale atmospheric model designed to simulate interactions between large-scale waves and small-scale activity, ideal for testing "subgrid-scale closure."

Key Results

  • Data Efficiency: The hybrid approach acts as a force multiplier. By starting with the "decent guess" provided by the physics model, CHyPP matched the performance of a massive single-reservoir model while using 150 times less training data.
  • Scalability: In KS equation tests, prediction error remained nearly constant even as the system size was scaled up, validating the parallel architecture.
  • Subgrid-Scale Closure: On the Lorenz III model, CHyPP effectively learned the statistical effect of unresolved turbulence on large-scale flow. Even when grid resolution was reduced to 1/8th of the original, CHyPP's prediction quality remained robust, whereas pure ML performance plummeted.
  • Reduced Overhead: Because the physics model provides global context, the individual parallel ML units require smaller overlap regions, significantly saving memory and communication bandwidth.

Conclusion and Future Work

CHyPP demonstrates that combining imperfect physical knowledge with parallelized machine learning can overcome the limitations of either approach used in isolation. While currently tested on 1D systems, the framework is a prime candidate for 3D global atmospheric forecasting and implementation on specialized AI hardware like FPGAs. Future research will focus on applying these methods to more complex, spatially inhomogeneous real-world environments.

Summary updated

Abstract

We consider the commonly encountered situation (e.g., in weather forecasting) where the goal is to predict the time evolution of a large, spatiotemporally chaotic dynamical system when we have access to both time series data of previous system states and an imperfect model of the full system dynamics. Specifically, we attempt to utilize machine learning as the essential tool for integrating the use of past data into predictions. In order to facilitate scalability to the common scenario of interest where the spatiotemporally chaotic system is very large and complex, we propose combining two approaches:(i) a parallel machine learning prediction scheme; and (ii) a hybrid technique, for a composite prediction system composed of a knowledge-based component and a machine-learning-based component. We demonstrate that not only can this method combining (i) and (ii) be scaled to give excellent performance for very large systems, but also that the length of time series data needed to train our multiple, parallel machine learning components is dramatically less than that necessary without parallelization. Furthermore, considering cases where computational realization of the knowledge-based component does not resolve subgrid-scale processes, our scheme is able to use training data to incorporate the effect of the unresolved short-scale dynamics upon the resolved longer-scale dynamics ("subgrid-scale closure").
BibTeX
@misc{wikner2020combining,
  author = {Alexander Wikner and Jaideep Pathak and Brian R. Hunt and Michelle Girvan and Troy Arcomano and Istvan Szunyogh and Andrew Pomerance and Edward Ott},
  title = {{Combining Machine Learning with Knowledge-Based Modeling for Scalable Forecasting and Subgrid-Scale Closure of Large, Complex, Spatiotemporal Systems}},
  year = {2020},
  eprint = {2002.05514},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.05514},
}
Figure