Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization

Simon Sinong Zhan, Qingyuan Wu, Frank Yang, Xiangyu Shi, Chao Huang, Qi Zhu, Zhu, Qi

Abstract

Offline-to-online deployment of reinforcement-learning (RL) agents must bridge two gaps: (1) the sim-to-real gap, where real systems add latency and other imperfections not present in simulation, and (2) the interaction gap, where policies trained purely offline face out-of-distribution states during online execution because gathering new interaction data is costly or risky. Agents therefore have to generalize from static, delay-free datasets to dynamic, delay-prone environments. Standard offline RL learns from delay-free logs yet must act under delays that break the Markov assumption and hurt performance. We introduce DT-CORL (Delay-Transformer belief policy Constrained Offline RL), an offline-RL framework built to cope with delayed dynamics at deployment. DT-CORL (i) produces delay-robust actions with a transformer-based belief predictor even though it never sees delayed observations during training, and (ii) is markedly more sample-efficient than naïve history-augmentation baselines. Experiments on D4RL benchmarks with several delay settings show that DT-CORL consistently outperforms both history-augmentation and vanilla belief-based methods, narrowing the sim-to-real latency gap while preserving data efficiency.

Introduction

In reinforcement learning (RL), the Markov property assumes the current observation provides sufficient information for decision-making. However, real-world systems frequently suffer from observation and actuation delays due to communication lags and computation times. By the time an agent receives a sensor reading, the environment's state has already changed. These delays break the Markov assumption, leading to significant performance degradation or system failure.

This challenge is particularly acute in offline RL, where agents are trained on static, often delay-free datasets (e.g., from simulators) but must be deployed in delay-prone real-world environments. Existing solutions like state augmentation (stacking past observations) often lead to high-dimensional spaces that are difficult to learn efficiently. Alternatively, training separate belief models can lead to compounding errors where small prediction inaccuracies accumulate over time.

Methodology: DT-CORL

The proposed framework, DT-CORL (Delay-aware Transformer-belief Constrained Offline RL), addresses these issues through a unified architecture that bridges the gap between delay-free training and delay-prone execution.

The DT-CORL architecture separates offline training on delay-free data from online adaptation in delayed environments using a Transformer-based belief model.

The architecture consists of three core components:

  1. Transformer-Based Belief Model: DT-CORL utilizes a Transformer to process the history of delayed observations and actions. It infers the current true state of the world, balancing the multi-step prediction accuracy needed for long horizons with the low latency required for real-time control.
  2. Joint Belief-Policy Optimization: Unlike two-stage approaches that train components in isolation, DT-CORL jointly optimizes the belief model, the actor (policy), and the critic (value function). This ensures the critic evaluates actions based on the specific latent representations produced by the belief model, minimizing distribution mismatch.
  3. Offline Constraints: To ensure safety and reliability, the framework incorporates policy constraints to prevent the agent from selecting out-of-distribution actions not supported by the training dataset.

Experimental Evaluation

The researchers evaluated DT-CORL using the D4RL benchmark across locomotion (Hopper, HalfCheetah, Walker2d), goal-conditioned (AntMaze), and dexterous manipulation tasks. Environments were modified to simulate deterministic and stochastic delays ranging from 4 to 64 steps.

Performance comparison on D4RL locomotion tasks under deterministic and stochastic delays. DT-CORL consistently outperforms standard and belief-based baselines.

Key Results

  • Superior Robustness: DT-CORL consistently outperformed state-of-the-art baselines. In the AntMaze "umaze" task with a 16-step delay, DT-CORL achieved a score of 40.0, while standard methods like CQL dropped to 12.7.
  • Locomotion Stability: On the Hopper-medium task with a 16-step delay, DT-CORL maintained a score of 71.8, significantly higher than the 42.9 achieved by belief-based CQL.
  • Long-Horizon Scaling: The model remained functional even at extreme delays of 64 steps, where most competing methods failed entirely.
  • Generalization to Stochasticity: Despite being trained with fixed delay assumptions, the Transformer-based representation generalized well to randomly varying delays, suggesting it learns robust temporal features.

Conclusion and Future Work

DT-CORL demonstrates that joint optimization of a Transformer-based belief model and a constrained policy can effectively bridge the sim-to-real latency gap in offline RL. While highly effective, future work could address heterogeneous delays across different sensors and extend the framework to high-dimensional visual inputs. Additionally, adapting to unknown or varying maximum delay lengths remains an open challenge for further research.

Summary updated

Abstract

Offline-to-online deployment of reinforcement-learning (RL) agents must bridge two gaps: (1) the sim-to-real gap, where real systems add latency and other imperfections not present in simulation, and (2) the interaction gap, where policies trained purely offline face out-of-distribution states during online execution because gathering new interaction data is costly or risky. Agents therefore have to generalize from static, delay-free datasets to dynamic, delay-prone environments. Standard offline RL learns from delay-free logs yet must act under delays that break the Markov assumption and hurt performance. We introduce DT-CORL (Delay-Transformer belief policy Constrained Offline RL), an offline-RL framework built to cope with delayed dynamics at deployment. DT-CORL (i) produces delay-robust actions with a transformer-based belief predictor even though it never sees delayed observations during training, and (ii) is markedly more sample-efficient than naïve history-augmentation baselines. Experiments on D4RL benchmarks with several delay settings show that DT-CORL consistently outperforms both history-augmentation and vanilla belief-based methods, narrowing the sim-to-real latency gap while preserving data efficiency.
BibTeX
@inproceedings{zhan2026belief,
  author = {Simon Sinong Zhan and Qingyuan Wu and Frank Yang and Xiangyu Shi and Chao Huang and Qi Zhu and Zhu, Qi},
  title = {{Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=3C1U86DcW4},
}
Figure