Keeping the Distributional Action Gap Alive Through Memory

When a reinforcement learning agent acts very often, each individual action matters very little. Its effect on long-run reward shrinks with the time step, and value-based learners lose the ability to tell good actions from bad ones. Recent work showed how to rescue this in distributional RL, where the agent learns the whole distribution of future returns rather than only their average. The fix is to learn the difference between an action's return distribution and the policy's, rescaled by the square root of the time step. Every rescue so far assumes the agent sees the true state. Real high-frequency systems, such as trading on noisy prices or robots with unmeasured velocities, are only partially observed. The agent must therefore build its own state with a recurrent network (a model that carries a memory vector forward from step to step).

This note argues that memory reintroduces the collapse through a side door. If the recurrent memory vector jumps by a fixed amount each step, regardless of how much physical time has passed, then the noise in the learning target grows as the time step shrinks. That noise swamps the rescaled action signal at exactly the rate the rescaling was meant to remove. The proposal is to state this as a testable condition on the memory: per-step changes must shrink like the square root of the time step, as a true belief filter's do. We then show that encoders meeting the condition keep the distributional action gap across frequencies and encoders violating it lose the gap.

Core Claim

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning proves three things for a decision interval $h$. The Wasserstein distance between action-conditioned and policy return distributions shrinks as $O(h^{1/2})$. The scalar mean gap shrinks as $O(h)$. The correctly normalized object is the superiority distribution

$$S_h(z,a) = h^{-1/2}\big(Z(z,a) - Z(z,\pi(z))\big),$$

where $Z(z,a)$ is the random return from taking action $a$ in state $z$ and then following policy $\pi$. Their method, DSUP, learns it by quantile temporal-difference learning on full states.

In a partially observed problem, $z_t = \phi(o_{\le t})$ is a learned summary of the observation history. Learning bootstraps from $Z(z_{t+1},\cdot)$. Any jump $\Delta z_t = z_{t+1}-z_t$ that does not reflect real information therefore enters the target as noise of order $\lVert \Delta z_t\rVert$. The useful part of the target is only $O(h^{1/2})$ per step. Our condition is

$$\mathbb{E}\lVert z_{t+1}-z_t\rVert^2 = O(h) \quad \text{as } h \to 0.$$

The $O(h)$ form comes from the fact that an exact filter of a diffusion observed through continuous noisy measurements has increments with variance proportional to elapsed time.

The prediction follows directly. Under the condition, the ratio of target noise to superiority signal stays bounded as frequency rises. If increments stay of order one, as in an LSTM whose update size ignores elapsed time, the ratio grows like $h^{-1/2}$. In that case recurrent DSUP should degrade with frequency much as QR-DQN degrades without the rescaling.

The claim concerns the condition, not a particular architecture. Several existing units satisfy it by construction:

None of these papers applies its unit to reinforcement learning or to action values. In this proposal they serve as method-suppliers.

Research Program

The work has three parts.

Theory. We first make the argument above precise in a linear-Gaussian partially observed diffusion. There the optimal belief is a Kalman filter, and the noise-to-signal ratio of the superiority target can be computed in closed form for two encoders: one satisfying the increment condition and one with fixed-size updates. This extends the full-state analysis of the NeurIPS 2024 paper. It is the motivating role that paper plays.

Algorithm. Second, we build recurrent DSUP. A recurrent encoder feeds a value-distribution head and an $h^{-1/2}$-scaled superiority head. The value head follows the decomposition $Z = Z_V + h^{1/2} S$, mirroring how Making Deep Q-learning methods robust to time discretization splits value and advantage in the scalar case. Every remaining quantity is set in physical time, following the scaling rules of that paper and of Continuous Soft Actor-Critic:

  • the discount;
  • exploration noise, via an Ornstein-Uhlenbeck process;
  • the learning rate;
  • replay capacity;
  • the length of the sequence used for backpropagation through time and for warming up the memory before each update, measured in seconds rather than steps.

The Tallec paper explicitly flags recurrent memory and replay under $\delta t \to 0$ as open. We address that question directly.

Diagnostics. Third, we measure two quantities per encoder and per frequency. One is the empirical increment statistic $\mathbb{E}\lVert\Delta z\rVert^2/h$. The other is the noise-to-signal ratio of the critic's learning target, in the style Bridging Discrete and Continuous RL uses to diagnose DDPG and SAC. We test whether the first predicts the second, and whether both predict control performance.

Baselines

Naive reference. The do-nothing reference is QR-DQN-style quantile regression with an LSTM encoder and no superiority rescaling. The NeurIPS 2024 paper reports that feedforward QR-DQN "could no longer distinguish between action distributions" at high trading frequencies. We expect the recurrent version to fail at least as badly, and it anchors the bottom of every plot.

Strongest existing method. The strongest competitor on this exact question is DSUP itself, from Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning. We run it with a feedforward network on two inputs: the raw current observation, and the true state where the simulator exposes it. The true-state version is an oracle upper bound for what memory can recover.

No reported numerical result was found on Lacuna for DSUP on the trading task. The file gives only the qualitative statement of "superior consistency across frequencies from 1Hz to 35Hz", so we must reproduce DSUP ourselves. We will tune it with the same search budget, in trials per frequency, as our method.

Beating DSUP-on-observations means two things across 1–35 Hz:

  • higher CVaR of return, and a flatter curve of CVaR against frequency;
  • closing a substantial fraction of the gap to the true-state oracle.

Scalar comparison. For a fully observed scalar reference, Continuous Soft Actor-Critic reports median score on the VMAS Sampling task as $\delta t$ moves from 0.1 to 0.01. Standard MASAC drops from 0.66 to 0.43, and CMASAC goes from 0.84 to 0.92. We will reproduce these numbers before masking observations. That checks that our physical-time scaling matches theirs.

Key ablation. The ablation that isolates the proposal's ingredient holds everything fixed except the encoder's increment behavior. We compare a plain GRU against the same GRU with a time-scaled leaky update, $z_{t+1} = z_t + \tfrac{h}{\tau}(\tilde z_t - z_t)$ with gain shrinking in $h$, and against CRU and LMU. All variants use the same parameter count, replay contents, and tuning budget.

Why not just…?

Why not just stack recent frames into DSUP's feedforward network? A fixed stack of $k$ frames covers only $k h$ seconds, which vanishes at high frequency. A fairer version subsamples the stack to cover a fixed physical window, and this is a real threat when the hidden quantity is a short-lived one such as velocity. We include the physically-timed stack as a baseline. The proposal should win only on tasks whose hidden variable integrates information over long windows, such as slowly drifting volatility in trading. If the timed stack matches our method there too, the memory claim is wrong for these tasks, and we will report that.

Why not just use an LSTM and let it learn to hold its state? An LSTM can, in principle, learn forget gates near one and small input gains. If it does so reliably, the increment condition holds without any architectural change. The diagnostic settles this. If tuned LSTMs show $\mathbb{E}\lVert\Delta z\rVert^2/h$ bounded and match CRU, the contribution reduces to the diagnostic and the physical-time training recipe, which would still be useful. We expect instead that an LSTM tuned at 1 Hz and deployed or retrained at 35 Hz violates the condition. That expectation matches the discrete-model failures reported for irregular sparse sampling in Continuous-Time Piecewise-Linear Recurrent Neural Networks.

Why not just use scalar advantage updating with a recurrent encoder? If only the mean return matters, a recurrent version of the Tallec method or CT-DDPG may be enough. The increment argument applies to it as well, with an $O(h)$ signal, so the scalar case is the more fragile one. We therefore evaluate on risk-sensitive CVaR objectives, where a scalar method has no tail to optimize. We also run recurrent DAU on mean return, so the reader can see whether the distributional machinery earns its cost when risk is not at stake.

Why not use Neural ODE memory? Neural ODE memory satisfies the condition but pays solver costs. Modeling Irregular Time Series with Continuous Recurrent Units reports f-CRU training 2–10x faster per epoch than Neural ODE models. At 35 Hz, sequences over a fixed number of seconds are long. An ODE-RNN is included only at the two lowest frequencies, as a cost-accuracy check.

Proposed Experiments

We use three test beds.

Partially observed Brownian control. The first extends the NeurIPS 2024 1D Brownian task: the agent sees position through additive noise of fixed per-sample variance. The optimal filter is known here, so we can check the predicted $h^{-1/2}$ growth of noise-to-signal for fixed-update encoders against the bounded ratio for Kalman-like ones.

Partially observed trading. The second is the commodity option trading task at 1, 5, 10, 20, and 35 Hz. We hide the volatility features so they must be inferred from the price history. The metric is CVaR at the 10% level and mean return, using the paper's data setup.

Partially observed control. The third uses Gymnasium Pendulum-v1 and HalfCheetah-v5 with velocities removed, across a decade of step sizes. These are the tasks used by Tallec et al. and Cheng et al. HalfCheetah also probes the contact non-smoothness that the DAU analysis excludes.

All runs use at least ten seeds, physical-time hyperparameter scaling, and equal tuning budgets.

Decisive checks. Three results would decide the question:

  • Across all encoders and frequencies, the measured increment statistic should predict the target noise-to-signal ratio, and that ratio should predict the drop in CVaR. A rank correlation near zero would falsify the core claim.
  • The time-scaled leaky GRU should behave like CRU and LMU, while the plain GRU behaves like the naive baseline. This would confirm the increment condition, rather than any specific architecture, as the operative ingredient.
  • Recurrent DSUP should keep CVaR flat from 1 to 35 Hz on the trading task while DSUP on raw observations degrades. If both degrade equally, memory is not the bottleneck.