Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data

Sung Woo Park, Kyungjae Lee, Junseok KwonICLR 2022 · OpenReview

Introduction

The authors introduce the Neural Markov Controlled Stochastic Differential Equation (CSDE), a novel probabilistic framework that applies stochastic optimal control theory to model complex time-series data. By utilizing multiple "control agents" that specialize in specific time intervals and a training scheme based on dynamic programming, the model achieves state-of-the-art performance in reconstruction and prediction. Crucially, it does so without needing the heavy recurrent neural network (RNN) wrappers commonly found in existing continuous-time models.

The Challenge with Continuous-Time Models

Modeling complex, irregularly sampled time-series data—such as medical records or stock prices—is notoriously challenging. While Neural Ordinary Differential Equations (ODEs) and Stochastic Differential Equations (SDEs) offer a continuous-time way to represent these dynamics, they suffer from a major structural flaw: they typically only use the initial state to propagate the entire trajectory.

To circumvent this, researchers often wrap ODEs/SDEs inside RNNs (e.g., ODE-RNN) or encoder-decoder structures. However, the authors argue this does not solve the fundamental "learnability" issue of the underlying dynamical model; it merely adds architectural complexity. The core challenge remains: How can we build an efficient SDE model that utilizes the rich information of an entire observed sequence to generate accurate trajectories directly in the data space, rather than relying on latent-space shortcuts?

Methodology: Neural Markov Controlled SDE

The paper proposes treating the modeling of time series as a stochastic optimal control problem, introducing several key innovations:

  1. Temporal Privacy (TP): Instead of relying on one monolithic network to model the entire timeline, the authors introduce multiple "control agents." Each agent is assigned a specific sub-interval of time, allowing them to specialize in local dynamics.
  2. Markov Dynamic Programming (Markov-DP): To train these agents efficiently, the authors leverage dynamic programming. They decompose the global optimization problem into smaller sub-problems, where each agent learns to minimize a cost functional within its own time window.
  3. MFcond (Markov Forward Conditional) Loss: Standard models often fail on high-dimensional data because they are trained with "naive" initial conditions. MFcond allows the model to utilize multiple past observations to condition future predictions, ensuring the model stays on track during inference.
  4. MBcond (Markov Backward Conditional) Loss: Drawing from classical control theory, the authors use the Hamiltonian-Jacobi-Bellman (HJB) equation to derive a backward loss. This ensures that the control agents are theoretically optimal and provides a powerful regularization effect.
Network training with the MFcond loss and subsequent network inference.

Experimental Setup

The model was evaluated on four diverse real-world datasets:

  • PhysioNet: 8,000 multivariate clinical time series from ICU patients (35 features).
  • Speech Commands: Audio records of spoken words, pre-processed into Mel-frequency cepstrum coefficients.
  • Beijing Air-Quality: 6-dimensional environmental data recorded hourly.
  • S&P 500 Stock Market: 6-dimensional financial data sampled every 30 minutes.

Key Insights and Results

  • Divide and Conquer: The "Temporal Privacy" approach demonstrates that using multiple specialized agents is significantly more efficient than using a single complex agent, even when the total number of parameters is held constant.
  • Direct Data-Space Modeling: Unlike many "Latent SDE" models that map data to a hidden space, CSDE infers complex trajectories directly in the data space, reducing the need for extra encoder/decoder networks.
  • Reconstruction Performance: On the PhysioNet dataset, CSDE-TP achieved a Mean Squared Error (MSE) of 0.728, significantly outperforming the next best baseline, mTAND (0.887), and Latent SDE (2.403).
Reconstruction performance comparison on PhysioNet and Speech Commands datasets.
  • Prediction and Uncertainty: For future-point prediction on Air Quality data, CSDE-TP outperformed all baselines by a wide margin in both MSE and Negative Log-Likelihood (NLL). Furthermore, on the S&P 500 dataset, CSDE-TP achieved a Maximum Mean Discrepancy (MMD) of 94.79, nearly doubling the distribution-matching performance of the standard Latent SDE (190.11).
  • Ablation Study: Removing either the multiple-agent structure or the MBcond loss led to a sharp decline in performance, proving that both the architectural and theoretical contributions are necessary.

Limitations and Future Work

While highly effective, the current model assumes continuous dynamics. Real-world data often contains sudden "jumps" or discontinuous changes. To address this, the authors plan to extend the model to a Markov Itô-Lévy jump diffusion model. Additionally, while multiple agents improve efficiency, there is likely an upper bound to how many agents can be effectively coordinated before the overhead of the dynamic programming scheme becomes a bottleneck. Finally, the model requires careful tuning, as the balancing parameter between forward and backward losses is crucial for optimal performance.

Summary updated

Abstract

We propose a novel probabilistic framework for modeling stochastic dynamics with the rigorous use of stochastic optimal control theory. The proposed model called the neural Markov controlled stochastic differential equation (CSDE) overcomes the fundamental and structural limitations of conventional dynamical models by introducing the following two components: (1) Markov dynamic programming to efficiently train the proposed CSDE and (2) multi-conditional forward-backward losses to provide rich information for accurate inference and to assure theoretical optimality. We demonstrate that our dynamical model efficiently generates a complex time series in the data space without extra networks while showing comparable performance against existing model-based methods on several datasets.
BibTeX
@inproceedings{park2022neural,
  author = {Sung Woo Park and Kyungjae Lee and Junseok Kwon},
  title = {{Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=zp-ZdP024EH},
}
Figure