TL;DR
The paper introduces DT-CORL (Delay-Transformer belief policy Constrained Offline RL), a novel framework that trains delay-robust reinforcement learning policies using only static, delay-free offline datasets. By embedding a transformer-based belief model directly into a constrained policy-iteration loop, DT-CORL successfully bridges the sim-to-real latency gap without requiring unsafe or costly online data collection under delayed dynamics.
What Problem Does This Paper Address?
When moving reinforcement learning (RL) agents from idealized simulations to the real world, they frequently stumble into two major gaps:
- The Sim-to-Real Latency Gap: Real-world systems suffer from mechanical, computational, and communication delays. When an agent receives an observation or executes an action late, the standard Markov assumption (that the current state depends only on the immediate previous state and action) breaks down, causing severe performance drops.
- The Interaction Gap: Gathering online interaction data to learn these delayed dynamics directly is often too dangerous, expensive, or impractical (e.g., crashing a drone or mismanaging a high-frequency trading pipeline).
While large, delay-free datasets (from simulators or legacy controllers) are widely available, standard offline RL algorithms cannot generalize from these delay-free logs to delay-prone deployment environments. Existing methods either rely on state augmentation (stacking histories), which suffers from extreme sample inefficiency as delays grow, or decoupled belief models, which suffer from compounding prediction errors.
Methods
DT-CORL reformulates policy-constrained offline learning in delayed Markov Decision Processes (MDPs) as a standard MDP optimization problem. The architecture consists of two primary components trained in a unified framework:
- Transformer-Based Belief Model: Instead of raw history stacking, DT-CORL uses a causal transformer to compress past sequences of delayed observations and actions into a compact latent belief state, predicting the true "present" state.
- Joint Belief-Policy Optimization: Crucially, rather than using a naive two-stage pipeline (training a belief filter, freezing it, and then training a policy), DT-CORL trains the belief model and the policy jointly. This ensures that the critic evaluates actions on the exact latent states the policy will encounter, preventing value-target biases.
To prevent out-of-distribution (OOD) issues common in offline RL, the policy optimization step incorporates a behavior-regularization constraint (similar to TD3+BC and ReBRAC) that penalizes large deviations from the offline dataset's behavior policy.
Datasets
The authors evaluated DT-CORL on standard benchmarks from the D4RL suite, specifically focusing on:
- AntMaze tasks (umaze, umaze-diverse, medium-play, large-play) for goal-conditioned control.
- MuJoCo locomotion tasks (Hopper, HalfCheetah, Walker2d) across various data quality tiers (medium, med-expert, med-replay, expert).
- Dexterous Hand Manipulation suite (Adroit) (Pen, Door, Hammer) to test high-dimensional, contact-rich environments.
Key Insights and Ideas
- The Danger of Decoupling: A frozen, pretrained belief model introduces unmodeled residual noise. In an offline setting, this noise biases the critic's Q-values, leading to rapid performance collapse during deployment. Joint training allows the belief model to specialize to the value function's error landscape.
- Why Transformers Win Here: Through ablation studies, the authors found a clear trade-off in sequence modeling for belief states. Ensemble MLPs are lightweight but accumulate errors rapidly over long horizons. Diffusion models are highly accurate but too computationally expensive for real-time online inference. Transformers strike the perfect balance, offering stable long-horizon multi-step accuracy alongside fast inference speeds (~22 ms).
Key Results
DT-CORL was tested under both deterministic and stochastic observation delays ranging from 4 to 16 steps ($\Delta \in \{4, 8, 16\}$).
- AntMaze: Online delay-robust methods like DBPT-SAC completely failed (0.0 score) due to the offline data constraint. DT-CORL consistently outperformed state-augmentation baselines (Augmented-BC, Augmented-CQL, Augmented-COMBO), showing significantly less performance degradation as delay lengths increased.
- MuJoCo Locomotion: DT-CORL maintained high normalized returns even at a severe 16-step delay. For instance, on
Hopper-mediumwith a 16-step deterministic delay, DT-CORL achieved a score of 71.8, compared to Belief-CQL's 42.9 and IQL's 4.8.
- Stochastic Resilience: Under stochastic delays (where delay varies at each step), state-augmentation methods degraded sharply due to high variance. DT-CORL's belief-based approach actually capitalized on effectively shorter average delays, demonstrating robust generalization across unseen delay distributions (Gaussian, Exponential, Binomial).
Limitations and Future Work
- Assumed Delay Bounds: DT-CORL currently assumes a known, fixed maximum delay horizon ($\Delta$) during training to construct its history buffers. Extending the framework to handle completely unknown, time-varying, or heterogeneous delays remains an open challenge.
- High-Dimensional Perceptual Inputs: The current architecture operates on state-vector inputs. Scaling this approach to vision-based manipulation tasks will likely require integrating spatial attention mechanisms or structured world models to handle the increased dimensionality.
Summary updated