Most reinforcement learning (RL) algorithms are designed for a world that moves in discrete "ticks." In these environments, an agent observes a state, takes an action, and receives a reward before the next state is revealed. However, physical reality—from robotic limbs to chemical reactors—operates in continuous time.
When we try to apply standard discrete-time RL to these high-frequency environments by simply shrinking the time step ($\Delta t$), the math breaks down. As the interval between actions approaches zero, the influence of any single action on the long-term reward vanishes. This leads to a fundamental problem known as the collapse of the Q-function.
The Vanishing Action Gap
In a standard discrete-time setting, the action-value function $Q(s, a)$ represents the expected return of taking action $a$ and following a policy thereafter. As Corentin Tallec, Léonard Blier, and Yann Ollivier demonstrate in Making Deep Q-learning methods robust to time discretization, this formulation is fragile.
As the time step $\Delta t$ becomes infinitesimally small, the immediate reward $r \Delta t$ goes to zero, and the next state $s'$ becomes indistinguishable from the current state $s$. Mathematically, the Q-function $Q(s, a)$ converges to the state-value function $V(s)$ for all actions. If $Q(s, a_1) \approx Q(s, a_2) \approx V(s)$, the agent loses the "action gap"—the numerical difference that allows it to distinguish the optimal action from a sub-optimal one. In high-frequency control, the learning signal is effectively drowned out by environmental noise, a phenomenon Huimin Han and Shaolin Ji describe as the signal being lost to the system's quadratic variation in Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization.
Advantage Updating as a Solution
To prevent this collapse, we must change how we represent the value of an action. The seminal work by Mance E. Harmon, Leemon C. Baird III, and A. Harry Klopf in Advantage Updating Applied to a Differrential Game introduced Advantage Updating.
Instead of learning $Q(s, a)$ directly, the agent learns the state value $V(s)$ and a separate advantage term $A(s, a)$. Crucially, this advantage is scaled by the inverse of the time step ($1/\Delta t$). By normalizing the advantage in this way, the relative benefit of one action over another remains finite and detectable even as the time step approaches zero. Corentin Tallec modernized this for the deep learning era with Deep Advantage Updating (DAU), which separates the value and advantage heads to maintain robustness across any time discretization.
From Bellman to Hamilton-Jacobi-Bellman
While discrete RL relies on the Bellman equation, continuous-time RL is grounded in the Hamilton-Jacobi-Bellman (HJB) equation. In Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time, Insoon Yang and Jeongho Kim 0002 provide the rigorous foundation for this shift. They treat the Q-function as a "viscosity solution" to an HJB equation, allowing for model-free learning in continuous-state and continuous-action spaces without losing action-specific information.
Other researchers approach this by modeling the system dynamics as continuous flows. Çağatay Yıldız, Markus Heinonen, and Harri Lähdesmäki use Bayesian Neural Ordinary Differential Equations (ODEs) in Continuous-time Model-based Reinforcement Learning to create agents that are robust to irregularly sampled data—a common real-world problem where "ticks" don't happen at perfectly regular intervals.
Modern Frontiers in Continuous Control
The challenge of continuous-time collapse extends beyond simple expected values into the realm of distributions and diffusion models:
- Distributional RL: Harley Wiltzer and Marc G. Bellemare found that action-conditioned return distributions also collapse toward the mean policy return at high frequencies. In Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning, they introduce "superiority," a probabilistic version of the advantage function that preserves the distinction between action distributions.
- Diffusion Policies: In Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control, Chengxiu Hua and colleagues link the "score" of a diffusion policy to the action gradient of the value function. This allows the agent to navigate the continuous action space by following the gradient of the value function, bypassing the need for a discrete Q-table entirely.
- Operator Theory: For offline settings, Nicolas Hoischen and Petar Bevanda propose O-CTRL in Operator Models for Continuous-Time Offline Reinforcement Learning. They represent system dynamics as infinitesimal generators, providing a way to learn optimal control from historical data with theoretical guarantees on error bounds.
By moving away from discrete-time assumptions and embracing the underlying differential equations of the environment, these methods ensure that RL agents remain capable of precise, high-frequency decision-making in the physical world.