Red Teaming with Mind Reading: White-Box Adversarial Policies Against RL Agents

Stephen Casper, Taylor Killian, Gabriel Kreiman, Dylan Hadfield-Menell

TL;DR

This paper introduces "white-box" adversarial policies for Reinforcement Learning (RL), where an attacker can "read the mind" of a target agent by observing its internal latent states. The authors demonstrate that this internal access allows adversaries to identify vulnerabilities up to 10x more efficiently than traditional black-box methods in both competitive games and language models.

What Problem Does This Paper Address?

Identifying failure modes in AI systems before deployment is a critical safety challenge. In RL, this is often done through "red teaming"—training an adversarial agent to exploit a target agent. Traditionally, these attacks are "black-box," meaning the adversary only sees the environment state and treats the target as just another part of the world.

However, this ignores the rich internal structure of the target model. In supervised learning, white-box access (knowing the model's weights and gradients) is the gold standard for finding vulnerabilities. This paper explores whether a similar "white-box" advantage exists for RL, where the non-differentiable nature of environment interactions makes standard gradient-based attacks difficult to apply.

Methods

The authors propose a framework where the adversary’s observation is augmented with internal information ($m_t$) from the target agent at every timestep.

Figure 1: White-box adversarial policies. At each timestep, both the adversary (adv) and target (tgt) observe the state st. The adversary also observes information from the internal state of the target and concatenates this extra information, mt, into its observations. We demonstrate how these white- box adversarial policies are more useful than black-box ones for identifying vulnerabilities in a target model.

They test three types of internal information:

  1. Action & Value: The target's predicted action distribution and its value estimate ($V_{tgt}$).
  2. Latent Activations: The activations from the target's internal neural network layers.
  3. Full: A combination of both.

The adversary is trained using Proximal Policy Optimization (PPO). The authors apply this to two distinct domains:

  • Competitive Games: In the Google Research Football environment, the adversary uses the target's internal state to better predict and counter its moves.
  • Language Models (LMs): They introduce a novel RL-based red teaming method for GPT-2. Instead of changing the input text, the adversary learns to perturb the target's latent activations to force the model to generate toxic output, bypassing the non-differentiability of discrete text tokens.
Experimental setup for Gfootball and GPT-2

Datasets

  • Google Research Football (Gfootball): A physics-based 2-player soccer simulation used to train and test competitive agents.
  • GPT-2 (117M "small"): The target language model for toxicity attacks.
  • RoBERTa-based Toxicity Classifier: Used to provide the reward signal for the adversary during language model red teaming.
  • OpenAI Gym HalfCheetah: Used for experiments in Robust Adversarial Reinforcement Learning (RARL).

Key Insights and Ideas

  • The Value Function Shortcut: One of the most powerful insights is that giving an adversary the target's value estimate ($V_{tgt}$) makes learning much easier. Since the adversary's goal is to minimize the target's reward, their value functions are roughly opposites ($V_{adv} \approx -V_{tgt}$). This internal hint helps the adversary converge on a winning strategy far faster.
  • Bypassing Non-Differentiability: By treating latent perturbations in a language model as "actions" in an RL loop, the authors provide a general method for red teaming text generators against any reward function (e.g., toxicity, bias, or misinformation) without needing to differentiate through the token sampling process.
  • The Defender’s Advantage: The authors argue that white-box access is typically available to a model's creators but not to external malicious actors. Therefore, these techniques primarily empower developers to find and fix flaws more effectively than an external attacker could.

Key Results

  • 10x Speedup in Discovery: In the Gfootball environment, white-box adversaries achieved the same performance in just 5 million timesteps that black-box controls required 50 million timesteps to reach.
  • Higher Asymptotic Performance: White-box attacks didn't just learn faster; they ultimately found more effective exploits, resulting in significantly higher net points per game.
Figure 3: Results for white-box adversarial policy attacks, measured in net points per game. (a) Performance curves for Gfootball target agents. The curves give the mean and standard error of the mean across n = 20 target agents. The first 25 million timesteps of training is against a rule-based “bot,” and the action entropy is rewarded while the second 25 million timesteps is against a peer and the action entropy is penalized. (b) Performance curves over 50 million timesteps for various adversarial attackers against the target agents. Notably, the best white-box adversarial policies do as well after 5 million timesteps as the black-box control does after 50 million. Curves give the mean and standard error of the mean across n = 20 targets. Three p values are shown below giving the results of a one-sided t test for the hypothesis that each white-box policy beats the black-box control.
  • Effective LM Red Teaming: White-box attacks on GPT-2 were more sample-efficient than black-box versions, successfully forcing the model to generate highly toxic completions from benign prompts.
Language model toxicity results

Limitations and Future Work

  • RARL Limitations: Interestingly, the authors found no clear benefit to using white-box adversaries for adversarial training (RARL) in simple control environments like HalfCheetah. The target agents did not become significantly more robust than those trained against black-box adversaries.
  • Modest LM Gains: While white-box access helped in LMs, the performance gap over black-box methods was less dramatic than in the gaming environment.
  • Future Directions: The authors suggest exploring "gray-box" attacks, where an adversary might learn a model of the target's internal state from black-box observations to gain some of the efficiency benefits of "mind reading" without direct access.

Summary updated

Abstract

Adversarial examples can be useful for identifying vulnerabilities in AI systems before they are deployed. In reinforcement learning (RL), adversarial policies can be developed by training an adversarial agent to minimize a target agent's rewards. Prior work has studied black-box versions of these attacks where the adversary only observes the world state and treats the target agent as any other part of the environment. However, this does not take into account additional structure in the problem. In this work, we study white-box adversarial policies and show that having access to a target agent's internal state can be useful for identifying its vulnerabilities. We make two contributions. (1) We introduce white-box adversarial policies where an attacker observes both a target's internal state and the world state at each timestep. We formulate ways of using these policies to attack agents in 2-player games and text-generating language models. (2) We demonstrate that these policies can achieve higher initial and asymptotic performance against a target agent than black-box controls. Code is available at https://github.com/thestephencasper/lm_white_box_attacks
BibTeX
@inproceedings{casper2023red,
  author = {Stephen Casper and Taylor Killian and Gabriel Kreiman and Dylan Hadfield-Menell},
  title = {{Red Teaming with Mind Reading: White-Box Adversarial Policies Against RL Agents}},
  booktitle = {SafeAI @ AAAI},
  year = {2023},
  url = {https://openreview.net/forum?id=AgMrFXbi63N},
}
Figure