Introduction
Reinforcement learning (RL) agents are highly vulnerable to data poisoning. In real-world deployments, an adversary might manipulate rewards or environment transitions—for instance, trolls intentionally misteaching a chatbot, or adversarial stickers causing an autonomous vehicle to misread a traffic sign.
This paper introduces Filtered Policy Gradient (FPG), the first RL algorithm that provides theoretical guarantees and strong empirical performance even when an adversary adaptively corrupts a constant fraction of training episodes.
The Vulnerability of Standard RL
Most RL algorithms assume the environment is "honest," meaning the rewards and state transitions the agent experiences are genuine. Existing research into "Robust RL" attempts to address poisoned data but often has major blind spots:
- Limited Scope: They frequently assume only rewards are corrupted, whereas a real adversary might control the transitions (the "next state") as well.
- Weak Adversaries: They often assume the number of corrupted episodes is tiny (sublinear) or that the adversary isn't "adaptive" (meaning it doesn't change its strategy based on the agent's actions).
- Lack of Scalability: Many solutions only work for simple "tabular" problems and fail in high-dimensional environments like robotics.
The authors tackle a "strong" adversary: one that knows the learning algorithm, observes the agent's actions, and can arbitrarily change both the reward and the next state for up to an $\epsilon$-fraction of all learning episodes.
Filtered Policy Gradient (FPG)
The paper builds on the Natural Policy Gradient (NPG) framework. Their approach unfolds in two main steps:
- Proving "Natural" Robustness: They first show that standard NPG is surprisingly resilient to bounded reward corruption. If the adversary can only give rewards within a fixed range, NPG still converges to a reasonably good policy.
- The FPG Algorithm: To handle unbounded corruption (where an adversary could send a reward to infinity to completely derail a gradient estimate), they propose Filtered Policy Gradient (FPG).
The core innovation of FPG is replacing the standard linear regression step in NPG with a robust version using the SEVER algorithm. SEVER works by:
- Calculating the gradients for all data points in a batch.
- Computing the top singular vector of the gradient matrix to find the direction of highest variance.
- Assigning "outlier scores" to data points based on how much they pull the gradient in that direction.
- Filtering out the most suspicious points before updating the policy.
For their experiments, the authors implemented a practical version of FPG based on Trust Region Policy Optimization (TRPO).
Key Insights
- Gradient Space is the Key: You don't need to know which specific rewards are "fake." By looking at the gradients of the loss function, you can identify which data points are trying to force the policy to change in a way that is inconsistent with the rest of the data.
- Dimension Independence: Crucially, the error bounds for FPG do not scale with the dimension of the problem. This means the algorithm remains robust even as the neural networks used for the policy get larger and more complex.
- The "Running Backward" Phenomenon: Without protection, an adaptive attack can be so effective that it tricks an agent into learning the exact opposite of the desired behavior. For example, in the HalfCheetah task, a standard agent under attack learns to run backward to minimize its perceived reward. FPG successfully ignores this malicious data to learn normal forward locomotion.
Empirical and Theoretical Results
Theoretical Bound: The authors prove that FPG can find an $O(\epsilon^{1/4})$-optimal policy. They also provide a lower bound showing that no algorithm can do better than $O(\epsilon)$, proving that some performance loss is inevitable under constant corruption.
Empirical Dominance: The authors evaluated FPG on the MuJoCo continuous control benchmarks (e.g., Swimmer, Hopper, Walker2d, HalfCheetah, Ant, Humanoid). In simulations with a 1% corruption level ($\epsilon = 0.01$), standard TRPO failed completely in every environment. FPG, however, maintained performance nearly identical to the "clean" version of the algorithm without any attack.
Limitations and Future Directions
- Exploration: The current theory assumes the agent starts with an "exploratory" distribution (it already knows how to reach most states). Solving the problem of robust exploration—where an adversary tries to prevent the agent from ever seeing certain parts of the environment—remains an open challenge.
- Tightening the Bounds: There is still a theoretical gap between the $O(\epsilon^{1/4})$ upper bound and the $O(\epsilon)$ lower bound.
- Computational Overhead: The SEVER filtering step requires computing the top eigenvalue of a large matrix, which can be slow for very large neural networks. Developing more efficient robust solvers is a priority for scaling this to even larger problems.
Summary updated