Reinforcement Learning for Continuous Time Control

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 217 papers25 conceptsfrozen 2026-10-04

Most reinforcement learning (RL) algorithms are designed for a world that moves in discrete "ticks." In these environments, an agent observes a state, takes an action, and receives a reward before the next state is revealed. However, physical reality—from robotic limbs to chemical reactors—operates in continuous time.

When we try to apply standard discrete-time RL to these high-frequency environments by simply shrinking the time step ($\Delta t$), the math breaks down. As the interval between actions approaches zero, the influence of any single action on the long-term reward vanishes. This leads to a fundamental problem known as the collapse of the Q-function.

The Vanishing Action Gap

In a standard discrete-time setting, the action-value function $Q(s, a)$ represents the expected return of taking action $a$ and following a policy thereafter. As Corentin Tallec, Léonard Blier, and Yann Ollivier demonstrate in Making Deep Q-learning methods robust to time discretization, this formulation is fragile.

As the time step $\Delta t$ becomes infinitesimally small, the immediate reward $r \Delta t$ goes to zero, and the next state $s'$ becomes indistinguishable from the current state $s$. Mathematically, the Q-function $Q(s, a)$ converges to the state-value function $V(s)$ for all actions. If $Q(s, a_1) \approx Q(s, a_2) \approx V(s)$, the agent loses the "action gap"—the numerical difference that allows it to distinguish the optimal action from a sub-optimal one. In high-frequency control, the learning signal is effectively drowned out by environmental noise, a phenomenon Huimin Han and Shaolin Ji describe as the signal being lost to the system's quadratic variation in Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization.

Advantage Updating as a Solution

To prevent this collapse, we must change how we represent the value of an action. The seminal work by Mance E. Harmon, Leemon C. Baird III, and A. Harry Klopf in Advantage Updating Applied to a Differrential Game introduced Advantage Updating.

Instead of learning $Q(s, a)$ directly, the agent learns the state value $V(s)$ and a separate advantage term $A(s, a)$. Crucially, this advantage is scaled by the inverse of the time step ($1/\Delta t$). By normalizing the advantage in this way, the relative benefit of one action over another remains finite and detectable even as the time step approaches zero. Corentin Tallec modernized this for the deep learning era with Deep Advantage Updating (DAU), which separates the value and advantage heads to maintain robustness across any time discretization.

From Bellman to Hamilton-Jacobi-Bellman

While discrete RL relies on the Bellman equation, continuous-time RL is grounded in the Hamilton-Jacobi-Bellman (HJB) equation. In Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time, Insoon Yang and Jeongho Kim 0002 provide the rigorous foundation for this shift. They treat the Q-function as a "viscosity solution" to an HJB equation, allowing for model-free learning in continuous-state and continuous-action spaces without losing action-specific information.

Other researchers approach this by modeling the system dynamics as continuous flows. Çağatay Yıldız, Markus Heinonen, and Harri Lähdesmäki use Bayesian Neural Ordinary Differential Equations (ODEs) in Continuous-time Model-based Reinforcement Learning to create agents that are robust to irregularly sampled data—a common real-world problem where "ticks" don't happen at perfectly regular intervals.

Modern Frontiers in Continuous Control

The challenge of continuous-time collapse extends beyond simple expected values into the realm of distributions and diffusion models:

By moving away from discrete-time assumptions and embracing the underlying differential equations of the environment, these methods ensure that RL agents remain capable of precise, high-frequency decision-making in the physical world.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 10 papers
  2. Version 1 · 2026-10-04 · 17 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_distributional_action_gap_collapse_f5426e93,
  title = {distributional action gap collapse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distributional-action-gap-collapse/rdr_f5426e9371ed40699a6792c2d7fdc2e6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reinforcement_learning_for_continuous_ti_f5426e93_v1,
  title = {Reinforcement Learning for Continuous Time Control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reinforcement-learning-for-continuous-time-control/rdr_f5426e9371ed40699a6792c2d7fdc2e6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}