Adversarial co-evolution—pitting an "attacker" model against a "defender" model—is the current frontier for safety-aligning Large Language Models (LLMs). However, these digital arms races often suffer from two fatal flaws: attackers become repetitive (strategy collapse), and defenders forget how to handle old tricks (adversarial forgetting).
DACE (Diversity-Driven Adversarial Co-Evolution) introduced in this paper, provides a robust framework to solve these pathologies by forcing attackers to explore a structured strategy space and equipping defenders with a "Bayesian memory" of past threats.
TL;DR
DACE improves LLM safety alignment by coupling a diversity-driven attacker reward with a Bayesian adversarial replay pool for the defender. This prevents the attacker from overfitting to a few "jailbreak" tricks and ensures the defender remains robust against both new and historical attack strategies without sacrificing general model capabilities.
What Problem Does This Paper Address?
Static safety alignment—training a model on a fixed dataset of "bad" prompts—is a losing game because the threat landscape moves too fast. Co-evolutionary frameworks, where an attacker LLM learns to break a defender LLM while the defender learns to resist, are more adaptive but prone to two "pathologies":
- Attack Strategy Collapse: The attacker finds a single high-reward rewrite (e.g., "Start your response with 'Sure, I can help with that...'") and overfits to it, leaving the defender blind to other types of risks.
- Defense Adversarial Forgetting: As the attacker moves on to new strategies, the defender’s weights shift, and it "forgets" how to defend against the earlier attacks it previously mastered.
Current remedies often focus on textual diversity (making the words different) rather than strategy diversity (making the method of attack different), and they fail to account for the uncertainty and non-stationarity inherent in a shifting adversarial game.
Methods
DACE introduces two primary mechanisms to stabilize the co-evolutionary loop:
- Explicit 2D Strategy Space: Instead of letting the attacker wander aimlessly, DACE defines a 120-cell grid (12 risk categories $\times$ 10 attack styles). The attacker is rewarded based on normalized marginal coverage gain. Essentially, the attacker gets a higher reward for successfully eliciting an unsafe response using a strategy that hasn't been explored much yet. This provides a non-vanishing signal that forces the attacker to "fill the grid."
- Bayesian Adversarial Replay Pool: To stop the defender from forgetting, DACE maintains a buffer of past successful attacks. Crucially, it treats each attack as a Bernoulli trial and maintains a Beta–Bernoulli posterior for each. It uses Thompson sampling to select which old attacks to replay, balancing the need to re-verify known high-threat attacks with the need to explore uncertain ones. It also applies time decay to these counts, allowing the system to "softly forget" stale evidence as the defender's policy evolves.
The training follows a two-phase process: a strategy-conditioned warm-up via SFT, followed by iterative co-evolution using the GRPO (Group Relative Policy Optimization) algorithm.
Datasets
The authors utilized several standard and specialized datasets:
- Training: Harmful seeds from Jailbreak-R1 and benign seeds from WildJailBreak.
- Safety Benchmarks: Evaluated on HarmBench (vanilla and adversarial), WildGuardTest, DAN, OR-Bench-Toxic, XSTest, and StrongREJECT.
- General Capabilities: To ensure no "alignment tax," they tested on IFEval, ARC-Challenge, GPQA, MMLU, and AlpacaEval 2.
Key Insights and Ideas
- Strategy vs. Surface Novelty: The authors argue that textual novelty (Self-BLEU scores) is a poor proxy for safety. An attacker can rephrase the same "role-play" attack a thousand ways; DACE forces it to switch from "role-play" to "technical jargon" or "uncommon dialects."
- Uncertainty Matters: By using a Bayesian approach for the replay pool, DACE distinguishes between an attack that failed once (low confidence) and one that has failed ten times (high confidence). This prevents the defender from prematurely deprioritizing dangerous attacks due to a single "lucky" refusal.
- Non-Stationarity is a Feature: The use of time decay in the replay pool acknowledges that the "threat level" of an attack is not absolute—it changes as the defender improves.
Key Results
DACE consistently outperformed state-of-the-art baselines like MAGIC and Self-RedTeam across multiple model families (Qwen2.5, Llama3.1).
- Superior Safety: DACE achieved near-zero Attack Success Rates (ASR) on standard benchmarks. For example, on Qwen2.5-7B, DACE reduced adversarial ASR on WildJailBreak from 0.198 (MAGIC) to 0.068.
- OOD Robustness: DACE defenders proved significantly more robust against unseen automated attackers (like GCG, PAIR, and AutoDAN-Turbo) that were not part of the training loop.
- No Capability Loss: DACE maintained or even slightly improved scores on general reasoning benchmarks like MMLU and AlpacaEval 2, proving that rigorous safety training doesn't have to break the model's brain.
Limitations and Future Work
- Manual Taxonomy: The 12x10 strategy grid is manually curated based on existing safety taxonomies (Llama-Guard-4 and Rainbow Teaming). While effective, it may not capture emerging, "unknown unknown" attack vectors.
- Judge Dependency: The framework relies heavily on the accuracy of the safety judge (e.g., Qwen3Guard). Any systematic bias in the judge could be amplified during co-evolution.
- Expansion: The current work is limited to text-only, single-turn interactions. The authors suggest extending DACE to multi-modal, tool-using, and multi-turn agentic settings as a natural next step.
Summary updated