TL;DR
This paper introduces "white-box" adversarial policies for Reinforcement Learning (RL), where an attacker can "read the mind" of a target agent by observing its internal latent states. The authors demonstrate that this internal access allows adversaries to identify vulnerabilities up to 10x more efficiently than traditional black-box methods in both competitive games and language models.
What Problem Does This Paper Address?
Identifying failure modes in AI systems before deployment is a critical safety challenge. In RL, this is often done through "red teaming"—training an adversarial agent to exploit a target agent. Traditionally, these attacks are "black-box," meaning the adversary only sees the environment state and treats the target as just another part of the world.
However, this ignores the rich internal structure of the target model. In supervised learning, white-box access (knowing the model's weights and gradients) is the gold standard for finding vulnerabilities. This paper explores whether a similar "white-box" advantage exists for RL, where the non-differentiable nature of environment interactions makes standard gradient-based attacks difficult to apply.
Methods
The authors propose a framework where the adversary’s observation is augmented with internal information ($m_t$) from the target agent at every timestep.
They test three types of internal information:
- Action & Value: The target's predicted action distribution and its value estimate ($V_{tgt}$).
- Latent Activations: The activations from the target's internal neural network layers.
- Full: A combination of both.
The adversary is trained using Proximal Policy Optimization (PPO). The authors apply this to two distinct domains:
- Competitive Games: In the Google Research Football environment, the adversary uses the target's internal state to better predict and counter its moves.
- Language Models (LMs): They introduce a novel RL-based red teaming method for GPT-2. Instead of changing the input text, the adversary learns to perturb the target's latent activations to force the model to generate toxic output, bypassing the non-differentiability of discrete text tokens.
Datasets
- Google Research Football (Gfootball): A physics-based 2-player soccer simulation used to train and test competitive agents.
- GPT-2 (117M "small"): The target language model for toxicity attacks.
- RoBERTa-based Toxicity Classifier: Used to provide the reward signal for the adversary during language model red teaming.
- OpenAI Gym HalfCheetah: Used for experiments in Robust Adversarial Reinforcement Learning (RARL).
Key Insights and Ideas
- The Value Function Shortcut: One of the most powerful insights is that giving an adversary the target's value estimate ($V_{tgt}$) makes learning much easier. Since the adversary's goal is to minimize the target's reward, their value functions are roughly opposites ($V_{adv} \approx -V_{tgt}$). This internal hint helps the adversary converge on a winning strategy far faster.
- Bypassing Non-Differentiability: By treating latent perturbations in a language model as "actions" in an RL loop, the authors provide a general method for red teaming text generators against any reward function (e.g., toxicity, bias, or misinformation) without needing to differentiate through the token sampling process.
- The Defender’s Advantage: The authors argue that white-box access is typically available to a model's creators but not to external malicious actors. Therefore, these techniques primarily empower developers to find and fix flaws more effectively than an external attacker could.
Key Results
- 10x Speedup in Discovery: In the Gfootball environment, white-box adversaries achieved the same performance in just 5 million timesteps that black-box controls required 50 million timesteps to reach.
- Higher Asymptotic Performance: White-box attacks didn't just learn faster; they ultimately found more effective exploits, resulting in significantly higher net points per game.
- Effective LM Red Teaming: White-box attacks on GPT-2 were more sample-efficient than black-box versions, successfully forcing the model to generate highly toxic completions from benign prompts.
Limitations and Future Work
- RARL Limitations: Interestingly, the authors found no clear benefit to using white-box adversaries for adversarial training (RARL) in simple control environments like HalfCheetah. The target agents did not become significantly more robust than those trained against black-box adversaries.
- Modest LM Gains: While white-box access helped in LMs, the performance gap over black-box methods was less dramatic than in the gaming environment.
- Future Directions: The authors suggest exploring "gray-box" attacks, where an adversary might learn a model of the target's internal state from black-box observations to gain some of the efficiency benefits of "mind reading" without direct access.
Summary updated