Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations

Shingo Ayabe, Takuto Otomo, Hiroshi Kera, Kazuhiko KawamotoarXiv 2024 · DOI · arXiv

Abstract

Offline Reinforcement Learning (RL) provides a pathway for training robotic controllers without the risks and costs associated with real-world trial and error. However, the transition from static datasets to dynamic environments often reveals a significant "reality gap." This paper investigates the robustness of offline RL agents against action-space perturbations—unexpected changes to physical outputs like actuator noise or failures. By evaluating popular offline algorithms (BCQ, TD3+BC, IQL) against online counterparts (PPO, SAC, TD3) on the D4RL benchmark, we demonstrate that offline agents are inherently more fragile. Our findings reveal that high-quality "expert" datasets can actually hinder robustness and that standard data augmentation techniques fail to provide the necessary corrective feedback required for resilient robot control.

Introduction

While Reinforcement Learning has shown immense promise in robot control, "online" learning remains impractical for expensive hardware due to the high risk of damage during exploration. Offline RL addresses this by training agents on pre-collected, static datasets. A critical challenge, however, is ensuring these agents can handle the unpredictability of the real world. While previous research has largely focused on sensor noise (state-space robustness), this study shifts the focus to action-space perturbations. We examine how offline agents respond when their physical commands are distorted by random noise or targeted adversarial attacks during deployment.

Methodology and Experimental Setup

The study utilizes the D4RL benchmark, focusing on locomotion tasks such as Hopper, HalfCheetah, and Ant. The researchers compared three offline RL algorithms—BCQ, TD3+BC, and IQL—against established online methods like PPO, SAC, and TD3.

Overview of the training and robustness evaluation pipeline, including the generation of adversarial perturbations using Differential Evolution.

To rigorously test robustness, three evaluation conditions were established:

  1. Normal: Baseline performance with no noise.
  2. Random: Uniformly distributed noise added to joint torque signals.
  3. Adversarial: A "worst-case" noise signal generated via Differential Evolution (DE), a black-box optimization technique that identifies perturbations that most effectively degrade performance without requiring environment gradients.

Key Insights: The Fragility of Offline Agents

The "Expert Trap"

A counterintuitive finding was that agents trained on "Expert" datasets often exhibited lower robustness than those trained on "Medium" datasets. Expert data is typically very narrow, showing only optimal behavior. In contrast, "Medium" datasets provide broader state-action coverage, effectively teaching the agent how to recover from sub-optimal states—a crucial skill when facing perturbations.

The Feedback Void

In online RL, agents receive immediate feedback from noisy actions, allowing them to learn corrective behaviors. In the offline setting, simply adding noise to a dataset is insufficient because the dataset lacks the resulting states and rewards for those perturbed actions. Without this closed-loop experience, the agent cannot learn to compensate for errors.

Results and Performance Analysis

The performance gap between offline and online agents becomes stark under adversarial conditions. As shown in the results table, offline agents frequently suffer catastrophic failure when faced with targeted perturbations.

Performance comparison of offline and online RL methods across different environments and perturbation conditions.

In the Hopper-v2 environment, an expert BCQ agent's reward dropped from 2371 to a mere 282 under adversarial attack. The Ant-v2 environment proved even more challenging; offline agents often yielded negative rewards, indicating a total loss of control, whereas online agents like PPO maintained significant performance (~2654). Furthermore, attempts to use "adversarial training" by injecting noise into the offline datasets yielded no significant improvement, confirming that the lack of outcome data for noisy actions is a fundamental bottleneck.

Conclusion and Future Directions

This research highlights that offline RL is fundamentally conservative and struggles when pushed into "unseen" states by physical perturbations. Robustness in the real world cannot be achieved through simple dataset augmentation alone.

Future work should explore:

  • Hybrid Offline-to-Online Learning: Using offline RL for pre-training followed by brief online fine-tuning to adapt to real-world distortions.
  • Robust Policy Ensembles: Developing models that combine multiple offline policies to stabilize behavior.
  • Recovery-Aware Data Collection: Intentionally including trajectories in training datasets that demonstrate recovery from failures and perturbations.

Summary updated

Abstract

Offline reinforcement learning, which learns solely from datasets without environmental interaction, has gained attention. This approach, similar to traditional online deep reinforcement learning, is particularly promising for robot control applications. Nevertheless, its robustness against real-world challenges, such as joint actuator faults in robots, remains a critical concern. This study evaluates the robustness of existing offline reinforcement learning methods using legged robots from OpenAI Gym based on average episodic rewards. For robustness evaluation, we simulate failures by incorporating both random and adversarial perturbations, representing worst-case scenarios, into the joint torque signals. Our experiments show that existing offline reinforcement learning methods exhibit significant vulnerabilities to these action perturbations and are more vulnerable than online reinforcement learning methods, highlighting the need for more robust approaches in this field.
BibTeX
@misc{ayabe2024robustness,
  author = {Shingo Ayabe and Takuto Otomo and Hiroshi Kera and Kazuhiko Kawamoto},
  title = {{Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations}},
  year = {2024},
  eprint = {2412.18781},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.18781},
}
Figure