Discretizing Continuous Control for Reinforcement Learning

27 concepts21 papers

Most physical systems—from turbulent fluid flows to industrial boilers—evolve continuously in time and space. However, the algorithms we use to control them, such as Reinforcement Learning (RL), typically operate in discrete steps. To bridge this gap, researchers have developed a rigorous framework for discretizing continuous-time stochastic differential equations (SDEs) into forms that discrete RL agents can solve without losing the underlying physics.

The Hamilton-Jacobi-Bellman Bridge

In continuous-time optimal control, the "gold standard" for finding the best action is the Hamilton-Jacobi-Bellman (HJB) equation. This partial differential equation describes how the optimal value function changes across space and time. The challenge is that the HJB equation is notoriously difficult to solve directly for complex systems.

Remi Munos and Paul Bourgine proposed that instead of solving the HJB equation analytically, we can treat it as a blueprint for a discrete Markov Decision Process (MDP). By applying finite-difference schemes to the HJB equation, the authors in Reinforcement Learning for Continuous Stochastic Control Problems showed that a continuous control problem can be transformed into a grid of discrete states and transitions. This allows standard RL algorithms to iteratively update a value function that, as the grid becomes finer, converges to the true continuous solution.

Stability through Monotonicity

Simply breaking a continuous system into a grid isn't enough; if the discretization is done poorly, the RL agent might diverge or find "optimal" paths that are physically impossible. This is often due to the loss of the Markov property during discretization.

To solve this, Remi Munos introduced the concept of viscosity solutions to the RL community. In A Convergent Reinforcement Learning Algorithm in the Continuous Case Based on a Finite Difference Method, the author proved that if a discretization scheme is "monotone," the RL algorithm is mathematically guaranteed to converge to the unique viscosity solution of the HJB equation.

Lingyi Yang recently extended this intuition in Q-Learning as a monotone scheme, reinterpreting Q-learning itself as a numerical solver. By viewing RL updates through the lens of numerical analysis, the author showed that stability in RL depends on the same mathematical conditions—consistency and monotonicity—required for traditional differential equation solvers to work.

From SDEs to Discrete Transitions

While the HJB equation governs the value, the underlying physics are often described by Stochastic Differential Equations (SDEs). To make these tractable for computers, researchers use numerical integrators like the Euler-Maruyama scheme.

This scheme acts as a translator:

  1. Continuous Physics: Nathalie Cauchi uses this to convert building automation models into discrete-time systems for formal verification in Benchmarks for stochastic models from building automation systems.
  2. Neural Architectures: Soon Hoe Lim et al. apply it to recurrent neural networks in Noisy Recurrent Neural Networks, providing a principled way to handle noise in sequential data.
  3. Sampling: Difan Zou et al. use it to implement Hamiltonian mechanics in computational sampling algorithms.

By discretizing the temporal evolution of a system, these methods allow agents to treat the "next state" as a stochastic transition in an MDP, even when the real-world system is a continuous flow.

Scaling to Complex Fields

The frontier of this research involves systems that are continuous not just in time, but in infinite-dimensional space, such as fluid dynamics modeled by Stochastic Partial Differential Equations (SPDEs).

In Deep Reinforcement Learning for Online Control of Stochastic Partial Differential Equations, Erfan Pirmorad et al. demonstrated that by discretizing both spatial and temporal dimensions, Deep RL agents can learn to damp out shock waves in turbulent systems. Similarly, Qian Qi proved in Universal Approximation Theorem of Deep Q-Networks that ResNet-style architectures can universally approximate the optimal functions of these continuous-time systems when framed through forward-backward SDEs.

Ultimately, this research direction shows that RL is not just a heuristic for games, but a powerful numerical method for solving the fundamental equations of motion and control in the physical world.

Key Foundational Work

Reinforcement Learning for Continuous Stochastic Control Problems (1997) — contributed 4 concept elements to this direction.
A convergent Reinforcement Learning algorithm in the continuous case based on a Finite Difference method (1997) — contributed 3 concept elements to this direction.
A Convergent Reinforcement Learning Algorithm in the Continuous Case Based on a Finite Difference Method (1997) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_differential_equation_discret_e4590789,
  title = {stochastic differential equation discretization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-differential-equation-discretization/rdr_e45907891d114640b8ef86e37c60b564},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_stochastic_differential_equation_discret_e4590789_v1,
  title = {stochastic differential equation discretization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-differential-equation-discretization/rdr_e45907891d114640b8ef86e37c60b564/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake