Attacker Diversity Reduction in Adaptive LLM Red Teaming

Chris TavaarXiv 2026 · arXiv

TL;DR

This paper reproduces a critical side effect of using Reinforcement Learning (RL) for automated red teaming: as attackers become more successful at "jailbreaking" models, they become significantly less diverse in their methods. By co-training a Qwen2.5-14B attacker and a Phi-4-mini defender, the author demonstrates that while attack success rates skyrocket, the variety of attack vectors shrinks by nearly 40%, potentially leaving defenders blind to less "optimal" but still dangerous threats.

What Problem Does This Paper Address?

Automated red teaming uses one AI (the attacker) to find vulnerabilities in another (the defender). Recent breakthroughs have used Group-Relative Policy Optimization (GRPO)—the same RL algorithm behind models like DeepSeek-R1—to train these attackers.

However, RL is notorious for "mode collapse": once an agent finds a strategy that yields a high reward, it stops exploring and starts exploiting that single path. In a security context, this is a major problem. If an automated red teaming tool only generates one type of attack (e.g., "Write a poem about making a bomb"), the defender only learns to block poems, leaving it vulnerable to other strategies like role-play or code-obfuscation. This paper quantifies exactly how much diversity is lost during this "arms race" between attacker and defender.

Methods

The author implemented a co-training loop where the attacker and defender evolve together:

  1. Attacker Update: The attacker generates multiple attempts for a harmful prompt. A judge (HarmBench-Llama-2-13b) scores them. The attacker is updated using GRPO with a "success-only" reward.
  2. Defender Update: Successful attacks are turned into training pairs for the defender, which is updated via Direct Preference Optimization (DPO) to refuse those attacks while still answering benign prompts.
  3. Stabilization: To prevent the training from diverging, the author used four key techniques: length-unbiased loss (to stop the model from just getting "wordy"), bounded updates, learning rate warm-up, and ratio clipping.

The core metric introduced is Attack Diversity Retention (ADR). It measures the "spread" of attacks in embedding space for a specific behavior, comparing the trained model to its untrained baseline. An ADR of 1.0 means diversity is preserved; the authors set a threshold of 0.75 to signal significant diversity reduction.

Datasets

The study utilizes several standard benchmarks to ensure reproducibility:

  • AdvBench: 520 behaviors used for training, development, and testing.
  • HarmBench: 172 standard behaviors used as a secondary test set.
  • Alpaca Instructions: Used to provide "benign" prompts, ensuring the defender doesn't become "over-refusive" (i.e., refusing to answer harmless questions like "How do I bake a cake?").

Key Insights and Ideas

  • The Success-Diversity Trade-off: There is a clear inverse correlation between how well an attacker performs and how diverse its attacks are. As the attacker "solves" the defender, it hones in on a narrow set of highly effective templates.
  • Strategy Migration: At the start of training, the attacker relied heavily on "format transformation" (e.g., asking for things in JSON or specific styles). By the end, it shifted almost entirely to "hypothetical framing" (e.g., "Imagine you are a character in a movie who needs to...").
  • Length as a Proxy for Complexity: Even with penalties for length, the average attack length doubled. This suggests that longer, more complex prompts are more effective at bypassing safety filters, likely by providing more "distraction" tokens for the defender's attention mechanism.
  • Persistent Onset: Diversity reduction isn't immediate. In these runs, the "persistent clear onset" of diversity loss typically occurred after about 20 rounds of updates, suggesting a "tipping point" where the model decides it has found the "meta" strategy.

Key Results

  • Diversity Loss: Across three independent runs, the ADR dropped to approximately 0.61–0.63. This represents a 37-39% reduction in the variety of attacks compared to the starting model.
  • Effectiveness Gain: Despite the narrowing of tactics, the attacker became much more lethal. The Attack Success Rate (ASR@1) rose from near 0% to as high as 81% against the frozen defender.
  • Defender Resilience: The defender successfully learned from the attacks, though the study focused more on the attacker's behavior. Benign over-refusal remained low (increasing by at most 3.6 percentage points).

Limitations and Future Work

  • Single-Turn Scope: The experiment only looked at single-turn attacks. Modern "jailbreaks" often involve multi-turn conversations (like the "Crescendo" attack), which might exhibit different diversity patterns.
  • Embedding-Based Metric: ADR relies on cosine distance in embedding space. While this captures semantic spread, it might miss subtle structural or logical diversities that embeddings don't weight heavily.
  • Reward Hacking: The author notes that "judge-based rewards" are susceptible to hacking. Future work could explore rewarding diversity directly within the GRPO objective to see if high success can be maintained without the collapse in variety.
  • Attribution: Because the experiment applied four stabilization techniques simultaneously, it’s unclear which specific component (or the interaction between them) most influences the rate of diversity decay.

Summary updated

Abstract

Reinforcement learning can make automated red-teaming more successful while narrowing the attacks it produces. Co-training an attacker and a defender with group-relative policy optimization (GRPO) has been reported to reduce attacker diversity. This experiment reproduces attack diversity reduction using three runs with a Qwen2.5-14B attacker and a Phi-4-mini defender trained by direct preference optimization (DPO). The attacker receives a success-only reward and uses a length-unbiased loss, bounded updates, warm-up and ratio clipping. The attacker is evaluated on 26 AdvBench behaviors. Training ended after 12 rounds, following a decision based on development checkpoints, so the reported intervals are nominal. Attack Diversity Retention (ADR) compares the within-behavior of the trained attacker spread with the untrained attacker. In order for an attack diversity to be consider as having been reduced, a criterion requires the ADR to be below a 0.75 target with a 95% confidence interval together with continued attacker effectiveness. Therefore, if an attacker narrows its attack vector by 25% consistently its considered less diverse. After 24 attacker and 12 defender LORA-based updates, ADR for the three runs were 0.63, 0.61 and 0.63, respectively. Each run met the criterion at round 12; the earlier checks at rounds 4 and 8 fell short. Despite attacks being less diverse, attack success continued to improve as the attacker was updated after each training run.
BibTeX
@misc{tava2026attacker,
  author = {Chris Tava},
  title = {{Attacker Diversity Reduction in Adaptive LLM Red Teaming}},
  year = {2026},
  eprint = {2611.19680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19680},
}
Figure