Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs

Jianzhun Du, Joseph Futoma, Finale Doshi-Velez

Abstract

This paper introduces a model-based reinforcement learning (RL) framework that leverages Neural Ordinary Differential Equations (ODEs) to master continuous-time environments. By modeling dynamics as a continuous flow rather than discrete steps, the authors achieve superior sample efficiency and the unique ability to optimize intervention timing. This approach is particularly effective for Semi-Markov Decision Processes (SMDPs) with irregularly sampled data, significantly reducing unnecessary interactions in domains like healthcare while maintaining high performance.

Introduction

Most deep RL successes, from Atari to Go, operate in discrete time: the agent observes the state, takes an action, and repeats at fixed intervals. However, the real world is continuous. In robotics, physics is governed by differential equations; in healthcare, patient data is often "irregularly sampled"—doctors take more measurements when a patient is crashing and fewer when they are stable.

Standard approaches usually "hack" this by discretizing time into even bins. If the bins are too large, you lose critical information about fast-moving dynamics. If they are too small, the planning horizon becomes impossibly long. While Semi-Markov Decision Processes (SMDPs) provide a mathematical framework for irregular timing, previous RL methods for SMDPs relied on simple linear models that couldn't capture complex, non-linear behaviors. This paper fills that gap by using Neural ODEs as the "world model" for RL.

Methods

The authors propose two primary architectures for modeling the environment's dynamics: the ODE-RNN, where the hidden state evolves according to an ODE between observations, and the Latent-ODE, an encoder-decoder structure where the entire trajectory is represented by an initial latent state that evolves continuously via an ODE.

Latent-ODE architecture for modeling continuous-time dynamics.

Key Innovations: * Action-Conditioned ODEs: The authors modified standard Neural ODEs to incorporate external actions ($a$) and time intervals ($\tau$), allowing the model to predict how an action taken now will affect the state at any arbitrary point in the future. * Joint Learning: The framework learns two things simultaneously: the state transition and the interval timing (when the next observation will occur). * Interval Optimization: Perhaps the most novel contribution is an algorithm to optimize the "interaction schedule." Instead of acting at every possible moment, the agent uses its internal ODE model to "skip" time steps where an intervention wouldn't significantly improve the long-term reward.

Key Results

The researchers evaluated their approach across several continuous-time domains, including Windy Gridworld, Acrobot, and an HIV Treatment Simulation. On MuJoCo tasks like the Swimmer and Hopper, the Latent-ODE approach developed high-performing policies in just 100k environment steps, while model-free baselines required 400k steps to reach the same level.

Reward curves comparing Latent-ODE against baselines on MuJoCo tasks.

Beyond sample efficiency, the ODE-based models consistently outperformed standard RNNs in predicting future states. When tested on time discretizations different from those seen during training, the Latent-ODE remained stable, whereas discrete-time baselines saw their performance collapse. In the HIV domain, the model learned to maintain patient health while reducing clinic visits by 75% by optimizing the interval timing.

Discussion and Conclusion

Because Neural ODEs model the underlying derivative of the state, they capture the "flow" of the environment, making them robust to changes in data collection frequency. This enables zero-shot time transfer, where a model trained on sparse data can plan for high-frequency schedules without retraining.

The primary bottleneck identified is computational cost. Training Latent-ODEs involves calling numerical ODE solvers, which is roughly 10 times slower than training a standard RNN. This limits current scalability to very high-dimensional state spaces. Future work could involve extending this framework to handle truly continuous action spaces and more complex hierarchical RL structures.

Summary updated

Abstract

We present two elegant solutions for modeling continuous-time dynamics, in a novel model-based reinforcement learning (RL) framework for semi-Markov decision processes (SMDPs), using neural ordinary differential equations (ODEs). Our models accurately characterize continuous-time dynamics and enable us to develop high-performing policies using a small amount of data. We also develop a model-based approach for optimizing time schedules to reduce interaction rates with the environment while maintaining the near-optimal performance, which is not possible for model-free methods. We experimentally demonstrate the efficacy of our methods across various continuous-time domains.
BibTeX
@inproceedings{du2020model,
  author = {Jianzhun Du and Joseph Futoma and Finale Doshi-Velez},
  title = {{Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=LRkmPxIzm-b},
}
Figure