White-Box Adversarial Policies in Deep Reinforcement Learning

Stephen Casper, Dylan Hadfield-Menell, Gabriel Kreiman

Abstract: In reinforcement learning (RL), adversarial policies can be developed by training an adversarial agent to minimize a target agent's rewards. Prior work has studied black-box versions of these attacks where the adversary only observes the world state and treats the target agent as any other part of the environment. However, this does not take into account additional structure in the problem. In this work, we take inspiration from the literature on white-box attacks to train more effective adversarial policies. We study white-box adversarial policies and show that having access to a target agent's internal state can be useful for identifying its vulnerabilities. We make two contributions: (1) We introduce white-box adversarial policies where an attacker observes both a target's internal state and the world state at each timestep. (2) We demonstrate that these policies can achieve higher initial and asymptotic performance against a target agent than black-box controls.

Introduction

Red teaming is a critical process in AI safety, aimed at identifying system failures before deployment. In the context of Reinforcement Learning (RL), this typically involves training an adversarial agent to minimize the target's reward. Traditional approaches have focused on black-box attacks, where the adversary treats the target as a static part of the environment, observing only the external world state $s_t$.

However, this approach overlooks the internal structure of the target model. This paper proposes White-Box Adversarial Policies, where the attacker has access to the target's internal "thoughts"—its latent activations and value judgments. This setup is particularly relevant for developers who possess full access to their models and wish to conduct rigorous stress testing.

Methodology

The core of the white-box framework is the observation vector $m_t$, which the adversary receives at each timestep $t$ alongside the environment state $s_t$. This vector $m_t$ can include action distributions (the target's intended next move), value estimates (the target's internal assessment of its current state), and latent activations (raw neural representations from the target's internal layers).

Architectural overview of white-box attacks in two-player games and language models.

The adversary is trained using Proximal Policy Optimization (PPO) to exploit these internal signals. The researchers evaluated this framework in two primary domains: Google Research Football (Gfootball), a physics-based soccer simulator, and GPT-2 (117M), where the adversary perturbs internal activations to induce toxic text generation.

Key Results

The experiments revealed a significant "mind-reading" advantage. In the Google Research Football environment, white-box adversaries demonstrated a 10x increase in sample efficiency, reaching the same performance level in 5 million steps that black-box adversaries required 50 million steps to achieve.

Performance comparison in Gfootball showing training progress and the superiority of white-box adversaries over black-box controls.

Key findings include:

  • Superior Asymptotic Performance: White-box policies consistently achieved higher net points per game than black-box controls.
  • Value Estimates as Critical Signals: The target's value estimate proved particularly useful; a high value for the target often serves as a direct signal for the adversary to pivot its strategy.
  • LM Efficiency: In GPT-2 toxicity experiments, white-box attacks were more sample-efficient, though the gains were more modest compared to the gaming domain.

Discussion and Conclusion

While white-box attacks are highly effective for red teaming, the study found that training against them (White-Box RARL) did not significantly improve robustness in simple continuous control tasks like HalfCheetah-v3 compared to black-box training. This suggests that white-box defense might require more complex environments or different training strategies to show clear benefits.

Future work includes exploring "theory of mind" approaches, where an adversary learns to infer the target's internal state from black-box observations, potentially extending these powerful white-box techniques to scenarios where internal access is restricted.

Code is available at: https://github.com/thestephencasper/lm_white_box_attacks

Summary updated

Abstract

In reinforcement learning (RL), adversarial policies can be developed by training an adversarial agent to minimize a target agent's rewards. Prior work has studied black-box versions of these attacks where the adversary only observes the world state and treats the target agent as any other part of the environment. However, this does not take into account additional structure in the problem. In this work, we take inspiration from the literature on white-box attacks to train more effective adversarial policies. We study white-box adversarial policies and show that having access to a target agent's internal state can be useful for identifying its vulnerabilities. We make two contributions. (1) We introduce white-box adversarial policies where an attacker observes both a target's internal state and the world state at each timestep. We formulate ways of using these policies to attack agents in 2-player games and text-generating language models. (2) We demonstrate that these policies can achieve higher initial and asymptotic performance against a target agent than black-box controls. Code is available at https://github.com/thestephencasper/lm_white_box_attacks
BibTeX
@inproceedings{casper2023white,
  author = {Stephen Casper and Dylan Hadfield-Menell and Gabriel Kreiman},
  title = {{White-Box Adversarial Policies in Deep Reinforcement Learning}},
  booktitle = {SafeAI @ AAAI},
  year = {2023},
  url = {https://openreview.net/forum?id=AgMrFXbi63N},
}
Figure