Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization

Simon Sinong Zhan, Qingyuan Wu, Philip Wang, Frank Yang, Xiangyu Shi, Chao Huang, Qi Zhu

TL;DR

The paper introduces DT-CORL (Delay-Transformer belief policy Constrained Offline RL), a novel framework that trains delay-robust reinforcement learning policies using only static, delay-free offline datasets. By embedding a transformer-based belief model directly into a constrained policy-iteration loop, DT-CORL successfully bridges the sim-to-real latency gap without requiring unsafe or costly online data collection under delayed dynamics.

What Problem Does This Paper Address?

When moving reinforcement learning (RL) agents from idealized simulations to the real world, they frequently stumble into two major gaps:

  1. The Sim-to-Real Latency Gap: Real-world systems suffer from mechanical, computational, and communication delays. When an agent receives an observation or executes an action late, the standard Markov assumption (that the current state depends only on the immediate previous state and action) breaks down, causing severe performance drops.
  2. The Interaction Gap: Gathering online interaction data to learn these delayed dynamics directly is often too dangerous, expensive, or impractical (e.g., crashing a drone or mismanaging a high-frequency trading pipeline).

While large, delay-free datasets (from simulators or legacy controllers) are widely available, standard offline RL algorithms cannot generalize from these delay-free logs to delay-prone deployment environments. Existing methods either rely on state augmentation (stacking histories), which suffers from extreme sample inefficiency as delays grow, or decoupled belief models, which suffer from compounding prediction errors.

Methods

DT-CORL reformulates policy-constrained offline learning in delayed Markov Decision Processes (MDPs) as a standard MDP optimization problem. The architecture consists of two primary components trained in a unified framework:

  • Transformer-Based Belief Model: Instead of raw history stacking, DT-CORL uses a causal transformer to compress past sequences of delayed observations and actions into a compact latent belief state, predicting the true "present" state.
  • Joint Belief-Policy Optimization: Crucially, rather than using a naive two-stage pipeline (training a belief filter, freezing it, and then training a policy), DT-CORL trains the belief model and the policy jointly. This ensures that the critic evaluates actions on the exact latent states the policy will encounter, preventing value-target biases.

To prevent out-of-distribution (OOD) issues common in offline RL, the policy optimization step incorporates a behavior-regularization constraint (similar to TD3+BC and ReBRAC) that penalizes large deviations from the offline dataset's behavior policy.

Figure 1: Overall pipeline of DT-CORL. In the Offline Training phase, trajectory data are augmented to train the transformer belief, and with the trained transformer belief, we conduct belief-based PI in the offline setting. In the Online Adaptation, we utilize the transformer belief to predict the current state from delayed observation, and adapt with offline-trained policy.

Datasets

The authors evaluated DT-CORL on standard benchmarks from the D4RL suite, specifically focusing on:

  • AntMaze tasks (umaze, umaze-diverse, medium-play, large-play) for goal-conditioned control.
  • MuJoCo locomotion tasks (Hopper, HalfCheetah, Walker2d) across various data quality tiers (medium, med-expert, med-replay, expert).
  • Dexterous Hand Manipulation suite (Adroit) (Pen, Door, Hammer) to test high-dimensional, contact-rich environments.

Key Insights and Ideas

  • The Danger of Decoupling: A frozen, pretrained belief model introduces unmodeled residual noise. In an offline setting, this noise biases the critic's Q-values, leading to rapid performance collapse during deployment. Joint training allows the belief model to specialize to the value function's error landscape.
  • Why Transformers Win Here: Through ablation studies, the authors found a clear trade-off in sequence modeling for belief states. Ensemble MLPs are lightweight but accumulate errors rapidly over long horizons. Diffusion models are highly accurate but too computationally expensive for real-time online inference. Transformers strike the perfect balance, offering stable long-horizon multi-step accuracy alongside fast inference speeds (~22 ms).

Key Results

DT-CORL was tested under both deterministic and stochastic observation delays ranging from 4 to 16 steps ($\Delta \in \{4, 8, 16\}$).

  • AntMaze: Online delay-robust methods like DBPT-SAC completely failed (0.0 score) due to the offline data constraint. DT-CORL consistently outperformed state-augmentation baselines (Augmented-BC, Augmented-CQL, Augmented-COMBO), showing significantly less performance degradation as delay lengths increased.
Table 1: Normalized returns (%) on D4RL AntMaze tasks under deterministic and stochastic observa- tion delays ∆∈{4, 8, 16}. Results are averaged over 3 seeds. Best per column (including ties) is shown in bold with a light blue background.
  • MuJoCo Locomotion: DT-CORL maintained high normalized returns even at a severe 16-step delay. For instance, on Hopper-medium with a 16-step deterministic delay, DT-CORL achieved a score of 71.8, compared to Belief-CQL's 42.9 and IQL's 4.8.
Table 2: DT-CORL vs. belief-based baselines (Belief-CQL, Belief-IQL) on D4RL MuJoCo tasks. Normalized returns (%). Delays: deterministic ∆∈{4, 8, 16}, stochastic ∆∼U(1, k), k ∈ {4, 8, 16}. Best results per column are shown in bold with light blue background.
  • Stochastic Resilience: Under stochastic delays (where delay varies at each step), state-augmentation methods degraded sharply due to high variance. DT-CORL's belief-based approach actually capitalized on effectively shorter average delays, demonstrating robust generalization across unseen delay distributions (Gaussian, Exponential, Binomial).

Limitations and Future Work

  • Assumed Delay Bounds: DT-CORL currently assumes a known, fixed maximum delay horizon ($\Delta$) during training to construct its history buffers. Extending the framework to handle completely unknown, time-varying, or heterogeneous delays remains an open challenge.
  • High-Dimensional Perceptual Inputs: The current architecture operates on state-vector inputs. Scaling this approach to vision-based manipulation tasks will likely require integrating spatial attention mechanisms or structured world models to handle the increased dimensionality.

Summary updated

Abstract

Offline-to-online deployment of reinforcement-learning (RL) agents must bridge two gaps: (1) the sim-to-real gap, where real systems add latency and other imperfections not present in simulation, and (2) the interaction gap, where policies trained purely offline face out-of-distribution states during online execution because gathering new interaction data is costly or risky. Agents therefore have to generalize from static, delay-free datasets to dynamic, delay-prone environments. Standard offline RL learns from delay-free logs yet must act under delays that break the Markov assumption and hurt performance. We introduce DT-CORL (Delay-Transformer belief policy Constrained Offline RL), an offline-RL framework built to cope with delayed dynamics at deployment. DT-CORL (i) produces delay-robust actions with a transformer-based belief predictor even though it never sees delayed observations during training, and (ii) is markedly more sample-efficient than naïve history-augmentation baselines. Experiments on D4RL benchmarks with several delay settings show that DT-CORL consistently outperforms both history-augmentation and vanilla belief-based methods, narrowing the sim-to-real latency gap while preserving data efficiency.
BibTeX
@inproceedings{zhan2026belief,
  author = {Simon Sinong Zhan and Qingyuan Wu and Philip Wang and Frank Yang and Xiangyu Shi and Chao Huang and Qi Zhu},
  title = {{Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=3C1U86DcW4},
}
Figure