Critical Sampling for Robust Evolution Operator Learning of Unknown Dynamical Systems

Ce Zhang, Kailiang Wu, Zhihai He

Abstract

Given an unknown dynamical system, what is the minimum number of samples needed for effective learning of its governing laws and accurate prediction of its future evolution behavior, and how to select these critical samples? In this work, we propose to explore this problem based on a design approach. Starting from a small initial set of samples, we adaptively discover critical samples to achieve increasingly accurate learning of the system evolution. One central challenge here is that we do not know the network modeling error since the ground-truth system state is unknown, which is however needed for critical sampling. To address this challenge, we introduce a multi-step reciprocal prediction network where forward and backward evolution networks are designed to learn the temporal evolution behavior in the forward and backward time directions, respectively. Very interestingly, we find that the desired network modeling error is highly correlated with the multi-step reciprocal prediction error, which can be directly computed from the current system state. This allows us to perform a dynamic selection of critical samples from regions with high network modeling errors for dynamical systems. Additionally, a joint spatial-temporal evolution network is introduced which incorporates spatial dynamics modeling into the temporal evolution prediction for robust learning of the system evolution operator with few samples. Our extensive experimental results demonstrate that our proposed method is able to dramatically reduce the number of samples needed for effective learning and accurate prediction of evolution behaviors of unknown dynamical systems by up to hundreds of times.

Introduction

Deep learning has emerged as a powerful paradigm for modeling dynamical systems, typically characterized by Ordinary Differential Equations (ODEs) or Partial Differential Equations (PDEs). However, standard data-driven approaches are notoriously data-hungry. For instance, modeling the chaotic 3D Lorenz system often requires a million data points to achieve acceptable accuracy. In real-world domains such as oceanography, climate science, or cardiovascular medicine, data collection is often expensive, difficult, or hazardous.

A fundamental challenge in efficient learning is the "chicken and egg" problem: to train a model effectively with minimal data, one must sample from regions where the model is most uncertain. However, calculating model error typically requires the "ground truth," which is exactly what the model is trying to learn. This work introduces a "Critical Sampling" framework to overcome this barrier by estimating error without ground truth.

Methodology

The proposed framework utilizes a three-pronged strategy to achieve robust learning with sparse data:

  1. Multi-Step Reciprocal Prediction: The core innovation involves training two networks simultaneously: a forward evolution network (predicting future states) and a backward evolution network (predicting past states). The authors provide a theoretical foundation showing that for autonomous systems, the backward evolution operator is mathematically equivalent to the forward operator of a time-reversed version of the same system. By performing a "round trip"—predicting $K$ steps forward and then $K$ steps backward—the system calculates a reciprocal prediction error based on the difference between the starting state and the returned state.
  2. Adaptive Critical Sampling: The framework iteratively identifies regions in the state space where the reciprocal error is highest using a peak-finding algorithm. New samples are then specifically collected from these "critical" areas to refine the model.
  3. Joint Spatial-Temporal Learning: To stabilize training under extreme data scarcity, a spatial dynamics network is integrated. This network learns local spatial patterns, allowing the system to augment its dataset by interpolating or predicting additional samples at unknown locations near existing data points.
Overview of the Critical Sampling framework for adaptive learning of unknown dynamical systems.

Key Insights: Error Correlation

The most significant finding of this research is that the reciprocal prediction error—which can be computed without ground-truth future states—serves as a highly reliable proxy for the actual network modeling error. As shown in Figure 5, there is a strong correlation between these two metrics across various dynamical systems and sample sizes. This discovery enables the model to identify its own failures and prioritize the acquisition of high-quality, informative samples over sheer data volume.

Correlation between reciprocal prediction error and actual network modeling error across different systems and sample sizes.

Experimental Results

The method was evaluated on four representative systems: the Damped Pendulum (2D ODE), a 2D Nonlinear ODE, the Lorenz System (3D chaotic), and the Viscous Burgers’ Equation (PDE).

  • Data Efficiency: On the Lorenz system, the Critical Sampling method reached the same error threshold as a uniform sampling baseline using only 1,765 samples compared to 1,000,000—a 566x reduction. The Damped Pendulum saw a 34x reduction (417 vs. 14,400 samples).
  • Comparative Performance: When compared to the state-of-the-art Markov Neural Operator (MNO) on the Lorenz system, the proposed method achieved lower error while requiring 45 times fewer training samples.
  • Inference Speed: While the iterative training and sampling process adds computational overhead, the final inference speed is identical to standard models, ensuring practicality for real-time applications.

Conclusion and Future Work

Critical Sampling demonstrates that the quality and strategic location of data points are far more critical than quantity for learning the laws of physics. While the current approach assumes the ability to sample at specific coordinates identified as critical, future work will address real-world constraints where system state control may be limited. Additionally, scaling the framework to higher-dimensional systems through reduced-order modeling or sparsity techniques remains a promising direction for future research.

Summary updated

Abstract

Given an unknown dynamical system, what is the minimum number of samples needed for effective learning of its governing laws and accurate prediction of its future evolution behavior, and how to select these critical samples? In this work, we propose to explore this problem based on a design approach. Starting from a small initial set of samples, we adaptively discover critical samples to achieve increasingly accurate learning of the system evolution. One central challenge here is that we do not know the network modeling error since the ground-truth system state is unknown, which is however needed for critical sampling. To address this challenge, we introduce a multi-step reciprocal prediction network where forward and backward evolution networks are designed to learn the temporal evolution behavior in the forward and backward time directions, respectively. Very interestingly, we find that the desired network modeling error is highly correlated with the multi-step reciprocal prediction error, which can be directly computed from the current system state. This allows us to perform a dynamic selection of critical samples from regions with high network modeling errors for dynamical systems. Additionally, a joint spatial-temporal evolution network is introduced which incorporates spatial dynamics modeling into the temporal evolution prediction for robust learning of the system evolution operator with few samples. Our extensive experimental results demonstrate that our proposed method is able to dramatically reduce the number of samples needed for effective learning and accurate prediction of evolution behaviors of unknown dynamical systems by up to hundreds of times.
BibTeX
@article{zhang2024critical,
  author = {Ce Zhang and Kailiang Wu and Zhihai He},
  title = {{Critical Sampling for Robust Evolution Operator Learning of Unknown Dynamical Systems}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2024},
  url = {https://openreview.net/forum?id=CZtMTBqh0R},
}
Figure