When evaluating the robustness of machine learning models, researchers often deploy automated agents to discover vulnerabilities. Whether finding pixel-level perturbations for vision models or generating jailbreak prompts for Large Language Models (LLMs), the goal is to find inputs that cause a target model to fail.
However, standard optimization algorithms suffer from a fundamental pathology: adversarial mode collapse (or strategy collapse). When an automated attacker is rewarded solely for maximizing its attack success rate, it quickly discovers a single, highly effective exploit. Once this vulnerability is found, the optimization pressure forces the attacker to generate minor variations of this same strategy rather than exploring the broader failure space.
As Dian Yu 0002 et al. demonstrated in Automatically Exposing Problems with Neural Dialog Models, reinforcement learning (RL) agents tasked with prompt generation frequently exploit the reward signal by converging on repetitive, universal triggers. This lack of diversity creates a false sense of security; a model might appear robust against an automated red-teaming suite simply because the suite is stuck in a narrow local optimum, leaving vast regions of the vulnerability surface completely unmapped. As Liwei Jiang notes in WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models, traditional automated methods fail to capture the wide variety of creative tactics used by real-world users in the wild.
To build truly robust systems, the research community has shifted from simply maximizing attack success to explicitly engineering adversarial strategy diversity.
Why Adversarial Optimization Collapses
To understand why adversarial generation collapses, we must look at the geometry of the optimization landscape.
In continuous spaces, such as image classification, untargeted adversarial attacks often gravitate toward a small set of easily reachable target categories. As Keiichiro Yamamura et al. explain in Diversified Adversarial Attacks based on Conjugate Gradient Method, standard gradient-based optimization methods like Projected Gradient Descent (PGD) frequently get trapped in the "narrow valleys" of a model's complex loss landscape, preventing the discovery of alternative misclassification classes.
In discrete spaces, like text generation, the problem is exacerbated by the categorical nature of language. Automated red-teaming tools often rely on fixed templates or narrow mutation operators, which limits linguistic diversity and prevents the discovery of sophisticated adversarial patterns like refusal suppression (Aman Goel et al. in TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practice).
When multi-turn interactions are optimized via RL, the agent typically experiences "reward overoptimization." It finds a specific tactical style or linguistic shortcut that bypasses the target's guardrails and repeatedly exploits it (Mingqian Feng et al. in SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attacks). This creates a fundamental trade-off between the success rate of an attack and the semantic diversity of the discovered vulnerabilities (Ninareh Mehrabi et al. in FLIRT: Feedback Loop In-context Red Teaming).
Four primary paradigms have emerged to break this bottleneck and force adversarial generators to explore the full failure space.
Paradigm 1: Relaxing Success to Prioritize Diversity
One powerful intuition for solving mode collapse is to change how we frame the optimization objective. If maximizing attack success forces the model into a narrow corner, what happens if we treat success as a baseline requirement instead of an optimization target?
This is the core insight behind the DiveR-CT framework proposed by Andrew Zhao et al. in DiveR-CT: Diversity-enhanced Red Teaming with Relaxing Constraints (and expanded in DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints). Instead of pushing the attack success score to its absolute maximum, the authors treat the safety score as a relaxed constraint.
Once a prompt is sufficiently successful at triggering a failure, the optimization objective shifts entirely toward maximizing semantic diversity, measured via a dynamic $k$-nearest neighbor ($k$-NN) distance in embedding space. By removing the pressure to over-optimize the success metric, the generator is free to explore a much wider spectrum of high-quality, non-repetitive attack vectors.
Paradigm 2: Intrinsic Motivation and Curiosity
Another approach borrows from reinforcement learning literature on sparse rewards: curiosity-driven exploration. Instead of relying solely on the external reward (whether the target model failed), the adversarial agent is given an intrinsic reward for generating novel inputs.
┌────────────────────────────────────────────────────────┐
│ Adversarial Agent │
└───────────────────────────┬────────────────────────────┘
│
│ Generates Prompt
▼
┌────────────────────────────────────────────────────────┐
│ Target Model │
└───────────────────────────┬────────────────────────────┘
│
│ Evaluates Response
▼
┌────────────────────────────────────────────────────────┐
│ Reward Engine │
│ ┌─────────────────────────────┐ │
│ │ Task Reward (Model Failure) │ │
│ └──────────────┬──────────────┘ │
│ │ │
│ ├─► Combined Reward ──► Updates Agent │
│ │ │
│ ┌──────────────┴──────────────┐ │
│ │ Intrinsic Curiosity Reward │ │
│ │ (Novelty / Embedding Distance) │
│ └─────────────────────────────┘ │
└────────────────────────────────────────────────────────┘
In Curiosity-driven Red-teaming for Large Language Models, Zhang-Wei Hong et al. introduce an auditor agent that receives an intrinsic reward proportional to the novelty of the generated prompt space. Similarly, Xiang Zheng et al. develop CALM (CALM: Curiosity-Driven Auditing for Large Language Models), which uses curiosity to uncover rare, long-tail vulnerabilities like private data leaks or subtle toxic completions. By rewarding the agent for exploring uncharted linguistic territory, curiosity-driven frameworks prevent the RL policy from collapsing into a single high-reward exploit, forcing it to map out thousands of unique vulnerabilities.
Paradigm 3: Geometrically and Algebraically Forcing Diversity
Rather than relying on intrinsic motivation, some frameworks explicitly penalize redundancy by incorporating geometric or algebraic constraints directly into the search step.
Explicit Diversity Penalties
In Query-Efficient Black-Box Red Teaming via Bayesian Optimization, Deokjae Lee et al. add a diversity penalty to the acquisition function of a Bayesian Optimization loop. This ensures that the algorithm balances the severity of a discovered failure with the conceptual distance from previously evaluated triggers, preventing the discovery of redundant failure cases.
Iterative Orthogonal Projection
Xiang Lisa Li et al. utilize an iterative diversity objective based on the Frank-Wolfe algorithm in Eliciting Language Model Behaviors with Investigator Agents. This mathematical formulation explicitly forces each new investigator agent to find "keys" (prompts) that are orthogonal or distinct from prior strategies, maximizing the coverage of the target model's behavior space.
Particle Repulsion
In the continuous domain, Thanh Nguyen-Duc et al. leverage Stein Variational Gradient Descent (SVGD) in Adversarial local distribution regularization for knowledge distillation. When attempting to generate multiple adversarial examples, standard random restarts often converge to the same local maxima. SVGD applies a repulsive force between multiple adversarial "particles" during optimization, preventing them from clustering together and providing a more comprehensive map of a model's decision boundaries.
Paradigm 4: Co-Evolutionary Self-Play
Perhaps the most dynamic solution to adversarial mode collapse is to turn the static target into a moving one. If a target model's defenses are fixed, an attacker can find a set of exploits and remain there. However, if the target continuously adapts to defend against those specific exploits, the old strategies lose their reward, forcing the attacker to innovate.
This co-evolutionary dynamic is framed as a two-player zero-sum game. Taeyoung Yun et al. implement this in Active Attacks: Red-teaming LLMs via Adaptive Environments by periodically safety-tuning the victim model during the attack training loop. As the victim patches its known vulnerabilities, the attacker is forced to discover increasingly diverse and subtle exploits to maintain its success rate.
┌──────────────────────────────┐
│ Initial Adversarial Agent │
└──────────────┬───────────────┘
│
▼ Finds Exploit A
┌──────────────────────────────┐
│ Static Target Model │
└──────────────┬───────────────┘
│
▼ Victim Safety-Tunes on A
┌──────────────────────────────┐
│ Adapted Target Model │
└──────────────┬───────────────┘
│
▼ Exploit A Fails; Forces Agent to Innovate
┌──────────────────────────────┐
│ Evolved Adversarial Agent │
└──────────────┬───────────────┘
│
▼ Finds Exploit B (New Strategy Mode)
Similarly, online self-play frameworks—such as those proposed by Mickel Liu et al. in Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models and Zizhao Wang et al. in Adversarial Reinforcement Learning for Large Language Model Agent Safety—demonstrate that co-training an attacker and a defender leads to a continuous expansion of the attack strategy space rather than a collapse. By chasing a moving target, the automated red-teaming system naturally uncovers a broad spectrum of vulnerabilities, ultimately producing a significantly more robust final model.
Key Foundational Work
Extending Robust Adversarial Reinforcement Learning Considering Adaptation and Diversity (2018) — contributed 1 concept elements to this direction.
Characterizing Bias in Classifiers using Generative Models (2019) — contributed 1 concept elements to this direction.