Abstract
Offline Reinforcement Learning (RL) provides a pathway for training robotic controllers without the risks and costs associated with real-world trial and error. However, the transition from static datasets to dynamic environments often reveals a significant "reality gap." This paper investigates the robustness of offline RL agents against action-space perturbations—unexpected changes to physical outputs like actuator noise or failures. By evaluating popular offline algorithms (BCQ, TD3+BC, IQL) against online counterparts (PPO, SAC, TD3) on the D4RL benchmark, we demonstrate that offline agents are inherently more fragile. Our findings reveal that high-quality "expert" datasets can actually hinder robustness and that standard data augmentation techniques fail to provide the necessary corrective feedback required for resilient robot control.
Introduction
While Reinforcement Learning has shown immense promise in robot control, "online" learning remains impractical for expensive hardware due to the high risk of damage during exploration. Offline RL addresses this by training agents on pre-collected, static datasets. A critical challenge, however, is ensuring these agents can handle the unpredictability of the real world. While previous research has largely focused on sensor noise (state-space robustness), this study shifts the focus to action-space perturbations. We examine how offline agents respond when their physical commands are distorted by random noise or targeted adversarial attacks during deployment.
Methodology and Experimental Setup
The study utilizes the D4RL benchmark, focusing on locomotion tasks such as Hopper, HalfCheetah, and Ant. The researchers compared three offline RL algorithms—BCQ, TD3+BC, and IQL—against established online methods like PPO, SAC, and TD3.
To rigorously test robustness, three evaluation conditions were established:
- Normal: Baseline performance with no noise.
- Random: Uniformly distributed noise added to joint torque signals.
- Adversarial: A "worst-case" noise signal generated via Differential Evolution (DE), a black-box optimization technique that identifies perturbations that most effectively degrade performance without requiring environment gradients.
Key Insights: The Fragility of Offline Agents
The "Expert Trap"
A counterintuitive finding was that agents trained on "Expert" datasets often exhibited lower robustness than those trained on "Medium" datasets. Expert data is typically very narrow, showing only optimal behavior. In contrast, "Medium" datasets provide broader state-action coverage, effectively teaching the agent how to recover from sub-optimal states—a crucial skill when facing perturbations.
The Feedback Void
In online RL, agents receive immediate feedback from noisy actions, allowing them to learn corrective behaviors. In the offline setting, simply adding noise to a dataset is insufficient because the dataset lacks the resulting states and rewards for those perturbed actions. Without this closed-loop experience, the agent cannot learn to compensate for errors.
Results and Performance Analysis
The performance gap between offline and online agents becomes stark under adversarial conditions. As shown in the results table, offline agents frequently suffer catastrophic failure when faced with targeted perturbations.
In the Hopper-v2 environment, an expert BCQ agent's reward dropped from 2371 to a mere 282 under adversarial attack. The Ant-v2 environment proved even more challenging; offline agents often yielded negative rewards, indicating a total loss of control, whereas online agents like PPO maintained significant performance (~2654). Furthermore, attempts to use "adversarial training" by injecting noise into the offline datasets yielded no significant improvement, confirming that the lack of outcome data for noisy actions is a fundamental bottleneck.
Conclusion and Future Directions
This research highlights that offline RL is fundamentally conservative and struggles when pushed into "unseen" states by physical perturbations. Robustness in the real world cannot be achieved through simple dataset augmentation alone.
Future work should explore:
- Hybrid Offline-to-Online Learning: Using offline RL for pre-training followed by brief online fine-tuning to adapt to real-world distortions.
- Robust Policy Ensembles: Developing models that combine multiple offline policies to stabilize behavior.
- Recovery-Aware Data Collection: Intentionally including trajectories in training datasets that demonstrate recovery from failures and perturbations.
Summary updated