Abstract
Adversarial examples can be useful for identifying vulnerabilities in AI systems before they are deployed. In reinforcement learning (RL), adversarial policies can be developed by training an adversarial agent to minimize a target agent's rewards. Prior work has studied black-box versions of these attacks where the adversary only observes the world state and treats the target agent as any other part of the environment. However, this does not take into account additional structure in the problem. In this work, we study white-box adversarial policies and show that having access to a target agent's internal state can be useful for identifying its vulnerabilities. We make two contributions. (1) We introduce white-box adversarial policies where an attacker observes both a target's internal state and the world state at each timestep. We formulate ways of using these policies to attack agents in 2-player games and text-generating language models. (2) We demonstrate that these policies can achieve higher initial and asymptotic performance against a target agent than black-box controls.
Introduction
As AI systems are increasingly integrated into high-stakes environments, "red teaming"—the systematic attempt to find flaws by intentionally inducing failure—becomes a critical safety practice. In the context of Reinforcement Learning (RL), red teaming typically involves training an adversarial agent to minimize the target agent's reward.
Traditionally, these adversarial policies have been developed under "black-box" assumptions: the adversary only observes the external world state, much like the target agent does. This approach, however, ignores the rich internal information available within the target's neural architecture. In supervised learning, white-box access to gradients significantly simplifies the discovery of adversarial examples. This research investigates whether a similar "white-box" advantage exists in RL, where gradients cannot be easily propagated through non-differentiable environments. By allowing an adversary to "read the mind" of its target—observing latent activations, value estimates, or action plans—can we more effectively identify vulnerabilities?
Methodology: White-Box Adversarial Policies
The core proposal is a framework where the adversarial policy, $\pi_{adv}$, receives two distinct inputs at each timestep $t$: the world state $s_t$ and a vector $m_t$ representing the target agent's internal state.
The researchers explored three variations of the internal state $m_t$:
- Action & Value: The target's current action distribution and its internal estimate of its own performance (value estimate).
- Latent: The raw activations from the target's internal neural network layers.
- Full: A concatenation of both action/value and latent information.
The adversary is trained using Proximal Policy Optimization (PPO) across two domains:
- Competitive Games: In the Google Research Football (Gfootball) environment, an adversary is trained to play soccer against a pre-trained target agent.
- Language Models: The researchers attacked GPT-2 (117M) by using RL to find perturbations in the model's latent space that compel it to generate toxic text, as measured by a RoBERTa-based toxicity classifier.
Key Results and Insights
The experiments demonstrate that internal information acts as a significant force multiplier for adversarial training.
Superior Performance and Efficiency in Games
In the Gfootball environment, white-box policies—particularly those leveraging latent activations—dramatically outperformed black-box controls. A key finding was the massive improvement in sample efficiency: white-box adversaries reached performance levels in just 5 million timesteps that required 50 million timesteps for black-box counterparts.
The inclusion of the target's value estimate proved particularly effective. Since the adversary's goal is the inverse of the target's, the target's value estimate ($V_{tgt}$) serves as a powerful heuristic for the adversary's own value function ($V_{adv} \approx -V_{tgt}$), accelerating the learning process.
Efficiency Gains in Language Models
For GPT-2, the white-box advantage was present but more modest. While both black-box and white-box attacks were eventually successful in forcing the model to generate toxic content, the white-box versions demonstrated higher sample efficiency, identifying effective latent perturbations more quickly. This suggests that while RL can bypass the non-differentiability of discrete text by operating in latent space, more sophisticated methods may be needed to fully exploit transformer internals.
Discussion: The Defender's Advantage
While "white-box" access might seem like a boon for attackers, the authors emphasize that it primarily empowers defenders. Developers naturally possess white-box access to their own models, whereas external malicious actors typically do not. Consequently, white-box red teaming serves as a powerful internal auditing tool, allowing developers to find and patch vulnerabilities that might otherwise remain hidden until a sophisticated attacker discovers them.
Conclusion and Future Work
This work establishes that white-box access to an RL agent's internal state significantly enhances the effectiveness of adversarial policies. Future research directions include "gray-box" attacks, where an adversary attempts to learn a model of the target's internal state from black-box observations, and exploring more advanced ways to leverage the complex internal structures of large-scale transformer models.
Summary updated