Robust policy gradient against strong data corruption

Xuezhou Zhang, Yiding Chen, Jerry Zhu, Wen Sun

TL;DR

This paper introduces a new algorithm, Filtered Policy Gradient (FPG), that makes reinforcement learning agents more resilient to adversarial attacks that can corrupt rewards and state transitions. FPG can learn effective policies even when a significant fraction of training data is maliciously altered, a crucial step for deploying RL in real-world, unpredictable environments.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn by interacting with an environment and receiving feedback in the form of rewards and state transitions. However, these signals can be noisy or even deliberately manipulated by an "adversary." Existing RL algorithms often struggle when faced with such data corruption, especially when the adversary can adapt its attacks throughout the learning process and corrupt both rewards and transitions. This vulnerability severely limits RL's application in safety-critical domains like autonomous driving or medical diagnosis, where data integrity is paramount. This paper tackles the challenge of building RL algorithms that are robust to strong, adaptive adversaries who can corrupt a significant portion of the training data.

Methods

The authors propose two main contributions:

  1. Natural Robustness of NPG: They first show that the standard Natural Policy Gradient (NPG) algorithm, surprisingly, exhibits some robustness to bounded adversarial rewards. If the adversary can only corrupt rewards within a certain range, NPG can still find a near-optimal policy.
  2. Filtered Policy Gradient (FPG): To handle more severe, unbounded corruption of both rewards and transitions, they introduce FPG. FPG is a robust variant of NPG that incorporates a robust linear regression subroutine (SEVER) to filter out potentially corrupted data points before they influence the policy update. This allows FPG to tolerate even unbounded corruption and achieve a meaningful learning guarantee.

Datasets

The experiments were conducted on the MuJoCo continuous control benchmarks. These are standard, high-dimensional simulated robotic control tasks used for evaluating RL algorithms. No new datasets were introduced.

Key Insights and Ideas

  • NPG's surprising resilience: It's unexpected that a standard algorithm like NPG shows inherent robustness to certain types of adversarial attacks, even without explicit modifications. This suggests that some existing RL methods might be more robust than previously thought.
  • The power of filtering: The core idea of FPG—identifying and discarding corrupted data points—is a powerful strategy for achieving robustness. By using a robust linear regression method, FPG effectively "cleans" the data before it's used to update the policy.
  • Adaptive adversaries are a serious threat: The paper emphasizes that their attack model is stronger than prior work because the adversary can adapt its strategy based on the agent's actions and the environment's feedback at each step. This makes the problem much harder and more realistic.
  • Theoretical guarantees for practical problems: The paper provides theoretical guarantees for FPG, showing it can learn effectively even with a significant fraction of corrupted data. This is crucial for building trust in RL systems deployed in the real world.

Key Results

  • Lower Bound: The paper establishes a theoretical lower bound, showing that no algorithm can guarantee finding a better than $O(\epsilon)$-optimal policy under their strong adversarial model, where $\epsilon$ is the fraction of corrupted episodes.
  • NPG Robustness: Under bounded reward corruption, NPG can find an $O(\sqrt{\epsilon})$-optimal policy.
  • FPG Performance: FPG can tolerate unbounded reward and transition corruption and find an $O(\epsilon^{1/4})$-optimal policy. This is a significant improvement, especially when $\epsilon$ is large.
  • Empirical Validation: A neural implementation of FPG on MuJoCo benchmarks demonstrated strong robust learning performance, outperforming standard algorithms like TRPO under adversarial attacks. Specifically, FPG learned the desired "running forward" policy while TRPO was fooled into learning a "running backward" policy in one experiment.

Limitations and Future Work

  • Exploration Assumption: FPG, like NPG, relies on an "exploratory initial distribution" for theoretical guarantees. Designing algorithms that achieve similar robustness without this assumption is an open question.
  • Tightness of Bounds: The theoretical bounds for FPG ($O(\epsilon^{1/4})$) and the lower bound ($O(\epsilon)$) are not tight. Finding the information-theoretically best possible robustness guarantee is an area for future research.
  • Computational Cost: The robust linear regression subroutine (SEVER) can be computationally expensive for very large neural networks. Developing more efficient robust learning methods is important for scaling FPG.
  • Generalization to Other RL Algorithms: The paper focuses on policy gradient methods. Investigating whether other RL algorithms (like PPO or SAC) can be robustified in similar ways is a promising direction.
  • Understanding Robustness: The authors suggest further exploring the connection between robustness to adversarial corruption and robustness to environment shifts or model misalignments.

Summary updated

Abstract

We study the problem of robust reinforcement learning under adversarial corruption on both rewards and transitions. Our attack model assumes an \textit{adaptive} adversary who can arbitrarily corrupt the reward and transition at every step within an episode, for at most $\epsilon$-fraction of the learning episodes. Our attack model is strictly stronger than those considered in prior works. Our first result shows that no algorithm can find a better than $O(\epsilon)$-optimal policy under our attack model. Next, we show that surprisingly the natural policy gradient (NPG) method retains a natural robustness property if the reward corruption is bounded, and can find an $O(\sqrt{\epsilon})$-optimal policy. Consequently, we develop a Filtered Policy Gradient (FPG) algorithm that can tolerate even unbounded reward corruption and can find an $O(\epsilon^{1/4})$-optimal policy. We emphasize that FPG is the first that can achieve a meaningful learning guarantee when a constant fraction of episodes are corrupted. Complimentary to the theoretical results, we show that a neural implementation of FPG achieves strong robust learning performance on the MuJoCo continuous control benchmarks.
BibTeX
@inproceedings{zhang2021robust,
  author = {Xuezhou Zhang and Yiding Chen and Jerry Zhu and Wen Sun},
  title = {{Robust policy gradient against strong data corruption}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=iNdfjd3ZKHx},
}
Figure