Continuous Time Reinforcement Learning for Physical Systems

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2137 papers162 conceptsfrozen 2026-10-04

Most reinforcement learning (RL) algorithms operate on a "ticking clock." They treat the world as a sequence of discrete snapshots—$s_t, a_t, r_t, s_{t+1}$—where the agent observes a state, takes an action, and jumps to the next state. This Markov Decision Process (MDP) framework has powered breakthroughs in games like Go or Atari, but it encounters a fundamental friction when applied to the physical world.

In robotics, finance, and chemical processing, time does not jump; it flows. Physical systems are governed by differential equations, not discrete transitions. When we force these continuous systems into discrete "ticks," we create a series of mathematical and practical hurdles that Kenji Doya first highlighted in Temporal Difference Learning in Continuous Time and Space.

The Discretization Trap

Standard RL agents are sensitive to their sampling frequency. If an agent is trained to control a robot at 10Hz, its learned policy often fails if the hardware is upgraded to 100Hz or if the sensor data arrives at irregular intervals.

As the time step $\Delta t$ approaches zero—the "high-frequency limit"—traditional discrete-time Bellman equations become numerically ill-conditioned. The difference between the current value $V(s_t)$ and the next value $V(s_{t+\Delta t})$ becomes infinitesimal, causing the signal to be drowned out by noise. As Lenart Treven and Andreas Krause argue in Efficient Exploration in Continuous-time Model-based Reinforcement Learning, discrete approximations often fail to capture the "inter-sample" dynamics, leading to unstable control and divergence from physical reality.

From Bellman to Hamilton-Jacobi-Bellman

To solve this, researchers are shifting from discrete Bellman equations to the Hamilton-Jacobi-Bellman (HJB) equation. If the Bellman equation is a recursive jump, the HJB is a partial differential equation that describes how value flows across a continuous state space.

In Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning, Harley E. Wiltzer and Marc G. Bellemare extend this to distributional RL. Instead of predicting a single expected return, they derive equations for the full distribution of future rewards in continuous time. This allows agents to account for the inherent stochasticity of systems governed by Stochastic Differential Equations (SDEs), where noise is not just a nuisance but a fundamental part of the system's evolution.

Robustness Through Temporal Invariance

The primary advantage of a continuous-time formulation is temporal invariance. Because the agent learns the underlying dynamics (the "flow") rather than a fixed-step transition, it remains robust to changes in sampling rates.

Lei Zhang and Ahmed H. Qureshi demonstrated this with VIP (Value Iteration via PINN) in Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning. By using Physics-Informed Neural Networks (PINNs) to solve the HJB equation, they created agents that maintain performance even when sensing and actuation frequencies fluctuate—a common occurrence in real-world hardware.

Solving the Action-Dependence Collapse

A long-standing mathematical hurdle in continuous-time RL is that the Q-function (which evaluates state-action pairs) often loses its dependence on the instantaneous action as the time step shrinks to zero. In the limit, the "next state" is essentially the "current state," making it hard to distinguish which action was better.

Recent work by Chengxiu Hua and colleagues in Continuous Q-Score Matching addresses this by linking RL to diffusion models. Their CQSM framework uses the "score" (the gradient of the log-density) of a diffusion policy to represent the action gradient of a Q-function. This allows for model-free, value-based learning in continuous action spaces without needing to discretize time or rely on a separate, brittle actor network.

Efficiency and Exploration

Continuous-time models also enable more sophisticated exploration. In Sample-efficient and Scalable Exploration in Continuous-Time RL, Klemens Iten and Florian Dorfler introduce COMBRL, which treats model uncertainty as an intrinsic reward. By operating in continuous time, the agent can decide not just what to do, but how long to do it, optimizing the "sensing" schedule to maximize information gain while minimizing expensive interactions.

Similarly, Mohamad Kazem Shirani Faradonbeh has shown in Thompson Sampling Efficiently Learns to Control Diffusion Processes that continuous-time frameworks allow for rigorous regret analysis. We can finally prove that an agent will learn to control an unknown physical system efficiently without the system "exploding" during the exploration phase.

The Path Forward

By treating time as a continuous dimension rather than a sequence of integers, this research direction bridges the gap between the fluid reality of physics and the rigid logic of computation. Whether through Soft HJB equations for maximum entropy exploration (Insoon Yang, Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control) or Operator Models that learn infinitesimal generators (Nicolas Hoischen, Operator Models for Continuous-Time Offline Reinforcement Learning), the goal is the same: agents that understand the flow of the world they inhabit.

Papers in this version

Showing 60 of 137 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 87 papers
  2. Version 1 · 2026-10-04 · 137 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_time_reinforcement_learning_578b0de7,
  title = {continuous-time reinforcement learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-time-reinforcement-learning/rdr_578b0de7d6df4007a32c344e8019922c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_continuous_time_reinforcement_learning_f_578b0de7_v1,
  title = {Continuous Time Reinforcement Learning for Physical Systems},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-time-reinforcement-learning-for-physical-systems/rdr_578b0de7d6df4007a32c344e8019922c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}